Batch file to upload only today's files to S3 (get filenames as list, loop through, execute cp command)

user_2616903 0
Viewed 1722

I need to create a script to copy only the files created today in Z:\ drive to AWS S3 using aws s3 cp command at a scheduler basis.

For example in Z:\, currently I have these files:

Filename                                            Date Modified
BELSIZE.0.DB2.NODE0000.CATN0000.20160313090011.001  3/13/2016 9:00 AM
BELSIZE.0.DB2.NODE0000.CATN0000.20160314090015.001  3/14/2016 9:00 AM
BELSIZE.0.DB2.NODE0000.CATN0000.20160315090010.001  3/15/2016 9:00 AM
BELSIZE.0.DB2.NODE0000.CATN0000.20160315100012.001  3/15/2016 10:00 AM

I need to copy only the last 2 files dated 3/15/2016 to AWS S3.

What I have now is:

  1. List the path + filename created today:

    forfiles /P "Z:\" /s /m *.* /d +0 /c "cmd /c echo @path

  2. Copy file to AWS S3: 

    aws s3 cp Z:\BELSIZE.0.DB2.NODE0000.CATN0000.20160315090010.001 s3://backup

I am thinking of getting 1) as a list, loop through 1) and execute 2). How can I do this?

I tried the below command. The file created today doesn't get copied to S3.

forfiles /P "Z:\" /s /m *.001 /d +0 /c "cmd /c aws s3 cp @path s3://backup"
aws-cli amazon-ec2 amazon-s3 batch-file
1 Answers

This is easy with WinSCP. To upload only today's files, use a batch file like:

winscp.com /log=S3.log /ini=nul /command ^
    "open s3://S3KEY:S3SECRET@s3.amazonaws.com/" ^
    "put -filemask=>=today Z:\* /bucket/" ^
    "exit"

The >=today keyword is supported by WinSCP 5.15 and newer only. In older versions, you can use %TIMESTAMP% syntax:

    "put -filemask=>=%%TIMESTAMP#yyyy-mm-dd%% Z:\* /bucket/" ^

You need to URL-encode special characters in the credentials. WinSCP GUI can generate an S3 script template, like the one above, for you.

Alternatively, since WinSCP 5.19, you can use -username and -password switches, which do not need any encoding:

    "open s3://s3.amazonaws.com/ -username=S3KEY -password=S3SECRET" ^

References:

(I'm the author of WinSCP)

user_8508480
Related