Azure Data Factory data flow file sink

user_8185525 0
Viewed 1870

I am using a .csv file to import data into an Azure SQL database. After the data import is complete I am now moving the source file from the Source container to myArchive container. I am now trying to save the filename as SaleData_yyyyMMdd_HHmm.csv, but, I have the folder with this name getting created and the file is broken down into multiple part files (part-00000-, part-00001-,...). Could you please guide me on how to specify the filename with current data & timestamp.

File System: myArchive

Folder Path: concat('SalesDepartment/Warehouse1/','SaleData_',toString(currentTimestamp(),'yyyyMMdd_HHmm'),'.csv')

enter image description here

azure-data-factory-pipeline azure-data-flow azure-data-factory-2 azure-data-factory
1 Answers

Folder path can be mentioned directly in the sink dataset. (Note, my source and sink both are delimited type)

For filename, Under sink data set, create a parameter to pass file name and use it in the file name portion of dataset.

SinkSetting

Use the below expression in copy activity sink's parameter value

@concat('SaleData_',formatDateTime(utcnow(),'yyyyMMdd_HHmm'),'.csv')

SinkParamValue

Remember, this just copies your source in a different name. We need to add a delete activity to delete the original source file.

If you are using a dataflow,

make sure you are choosing single partition in the optimize tab of Sink instead of Use current Partitioning.

Partitioningoption

Then, go to Settings, choose Output to SIngle file. Under filename, mention the expression with timestamp.

concat('SaleData_',toString(currentUTC('yyyyMMdd_HHmm')),'.csv')

Setting

user_149973040
Related