Databricks has a great Synapse connector that take a burden of bulk uploading. Under the hood, the connector is creating a temporary folder at your Blob Storage, and use Polybase external tables to load to Synapse very quickly.
Currently I have ETL doing bulk load to Synapse on daily basis (writting or overwritting single date partition).
Is there any option to ensure consistency while loading from Databricks to Synapse?
I mean partition switching or at least any transactional support. Shall I write a custom database cleansing logic for cases when Databricks job fall in the middle, so I have removed previous data and written only the half of dataset?
UPD
The official doc says "use the Data Source API". Is there any reasons of not loading data into main table except the risk to make updated partition inconsistent?
Apply some transformations to the data, then use the Data Source API to write the data back to another table in Azure Synapse.
df.write
.format("com.databricks.spark.sqldw")
.option("url", "jdbc:sqlserver://<the-rest-of-the-connection-string>")
.option("forwardSparkAzureStorageCredentials", "true")
.option("dbTable", "my_table_in_dw_copy")
.option("tempDir", "wasbs://...")
.save()