Azure data factory File Content Replace in the Azure blob

user_4128555 0
Viewed 192

Good morning, We have Azure Data Factory (ADF). We have 2 files that we want to merge into one another. The files are currently in the Azure Blob storage. Below are the contents of the files. We are trying to take the contents of File2.txt and replace the '***' in File1.txt. When finished, it should look like File3.txt.

File1.txt
OP01PAMTXXXX01997
***
CL9900161313

File2.txt
ZCBP04178      2017052520220525         
NENTA2340      2015033020220330         
NFF232174      2015052720220527

File3.txt
OP01PAMTXXXX01997
ZCBP04178      2017052520220525         
NENTA2340      2015033020220330         
NFF232174      2015052720220527
CL9900161313

Does anyone know how we can do this? I have been working with this for 2 days and it would seem that this should not be a difficult thing to do.

All the best, George

azure-data-factory-pipeline azure-data-factory-2 azure-data-factory
1 Answers

You can merge 2 files or more using ADF but i can't see a way where we can merge with a condition / control the way we merge files, so what i can recommend is to use Azure Function and do the merge programmatically. if you want to know how to merge files without preserving line priorities use my approach:

  1. create a pipeline
  2. add a "Copy activity"

in copy activity use this basic settings:

  • in source -> chose WildCard file path (select the folder that files are located at), make sure in wildcard path to write "*" in filename this will guarantee to chose all files under the same folder. this will merge all the files under the same folder.
  • in Sink -> make sure to select in Copy behavior Merge Files mode.

enter image description here

enter image description here

Output :

enter image description here

user_67888850
Related