I am having an issue with the command s3-dist-cp on Amazon EMR. What I want to achieve is being able to define the name of the file when merging all the little files in my S3 folder. Example:
s3://bucket/event/v1/2017/11/06/10/event-1234.json
s3://bucket/event/v1/2017/11/06/10/event-4567.json
s3://bucket/event/v1/2017/11/06/10/event-7890.json
.... so on
and the result is the following:
s3://test/test/event
I am able to merge all the files above but the result filename is wrong.
The command is:
s3-dist-cp --src s3://bucket/event/v1/2017/11/06/10/ --dest s3://test/test/ --groupBy='.*(event).*' --targetSize=2048
and the result I want to achieve is:
s3://test/test/events.hourly.json
How can I change the destination file name ?