Filter out file using wildcard path azure data factory

Question

I am working on a pipeline and while using the copy activity, in the file wildcard path I would like to skip a certain file and only copy the rest. Is there an expression for that ? I know that a * is used to match zero or more characters but in this case, I would like an expression to skip a certain file.

I am still very new to adf

Thank you.

I am still very new to adf

Thank you.

user_75838 · Answer

I'm not sure you can use the wildcard feature to skip a specific file, unless all the other files follow a pattern the exception does not follow.

One approach would be to use GetMetadata to list the files:

Note the inclusion of the "ChildItems" field, this will list all the items (Folders and Files) in the directory.

Next, use a Filter activity to reference only the files:

Items code:

@activity('Get Child Items').output.childItems

Filter code:

@and(equals(item().type,'File'),endswith(item().name,'.txt'))

NOTE: This example filters to Files with a .txt extension. You would change this code to meet your criteria.

Finally, use a ForEach to loop over the now filtered items. The ForEach would contain our COPY activity for each individual item:

user_15969107 · Answer

In Get Metadata activity, we can add an expression to get files of a specific pattern. I tried to write an expression to exclude files but was not successful.

Below is what I have tried to exclude/skip a file from the list of files to process.

Files in the Input folder:

Azure Data Factory Pipeline:

In my Input folder, I have 2 types of files .csv and .txt. You can add expression in the filename to get the only “.csv” files using Get Metadata activity