Data Factory Childitem modified or created date

user_13759008 0
Viewed 1440

I have a Data Factory V2 pipeline consisting of 'get metadata' and 'forEach' activities that reads a list of files on a file share (on-prem) and logs it in a database table. Currently, I'm only able to read file name, but would like to also retrieve the date modified and/or date created property of each file. Any help, please?

Thank you

azure-data-factory-2
2 Answers

According to the MS documentation. We can see File system and SFTP both support the lastModified property. But we only can get the lastModified of one file or folder at a time. enter image description here

I'm using File system to do the test. The process is basically the same as the previous post, we need to add a GetMetaData activity to the ForEach activity. This is my local files. enter image description here

  1. First, I created a table for logging.
create table Copy_Logs (
    Copy_File_Name varchar(max),
    Last_modified datetime
)

  1. In ADF, I'm using Child Items at Get Metadata1 activity to get the file list of the folder. enter image description here

  2. Then add dynamic content @activity('Get Metadata1').output.childItems at ForEach1 activity. enter image description here

  3. Inside the ForEach1 activity, using Last modified at Get Metadata2 activity. enter image description here

In the dataset of Get Metadata2 activity, I key in @item().name as follows. enter image description here

  1. Using CopyFiles_To_Azure activity to copy local files to the Azure Data Lake Storage V2. enter image description here I key in @item().name at the source dataset of CopyFiles_To_Azure activity. enter image description here

  2. At Create_Logs activity, I'm using the following sql to get the info we need. enter image description here

select '@{item().name}' as Copy_File_Name, '@{activity('Get Metadata2').output.lastModified}' as Last_modified
  1. In the end, sink to the sql table we created previously. The result is as follows. enter image description here
user_139794870

One way , I can think of is please add a new Getmetdata inside the FE loop and use paramterized dataset and pass a filename as the paramter . The below animation should helped , I did tested the same . HTH . enter image description here

user_111376790
Related