I have parquet files stored in Azure storage account and I need to filter them and copy them to delimited files. I can copy the files as a whole but I haven't figured out how to filter input files using Copy Activity. Input files are very simple, just couple of columns and filtering needs to be done based on values on one column. Here's an example to give you an idea:
Country | Municipality | City
cntry A | mncplty 1 | cty 1
cntry A | mncplty 1 | cty 2
cntry B | mncplty 1 | cty 1
cntry C | mncplty 1 | cty 1
cntry C | mncplty 2 | cty 1
cntry C | mncplty 2 | cty 2
I need to get every row from input files where Country = "cntry A".
If I were to do this in SQL I would do it something like this:
INSERT INTO delimited
SELECT * FROM parquet
WHERE Country = "cntry A";
How can I achieve this in Azure Data Factory? Can I do this just using Copy Activity or do I need to use some other activity?