Why Azure Databricks needs to store data in a temp storage in Azure

Admin 1
Viewed 334

I was following the tutorial about data transformation with azure databricks, and it says before loading data into azure synapse analytics, the data transformed by azure databricks would be saved on temp storage in azure blob storage first before loading into azure synapse analytics. Why the need to save it to a temp storage before loading into azure synapse analytics?

azure-synapse azure-databricks
1 Answers

The Azure storage container acts as an intermediary to store bulk data when reading from or writing to Azure Synapse. Spark connects to the storage container using one of the built-in connectors: Azure Blob storage or Azure Data Lake Storage (ADLS) Gen2.

The following architecture diagram shows how this is achieved with each HDFS bridge of the Data Movement Service (DMS) service on every Compute node connecting to an external resource such as Azure Blob Storage. PolyBase then bidirectionally transfers data between SQL Data Warehouse and the external resource providing the fast load performance.

enter image description here

Using PolyBase to extract, load and transform data The steps for implementing a PolyBase ELT for SQL Data Warehouse are:

  1. Extract the source data into text files.
  2. Load the data into Azure Blob storage, Hadoop, or Azure Data Lake Store.
  3. Import the data into SQL Data Warehouse staging tables using PolyBase.
  4. Transform the data(optional).
  5. Insert the data into production tables.
user_81884330
Related