We are getting several files in one s3 folder ( 130K files , combined size is 2GB ). Each file has Json data , could be one or many records. I need to merge these files into a single Json file and store it on s3. I don't want to download the files to local machine and then combine. Is there a way to do it using AWS SDK for Java ?
1 Answers
The simplest way to achieve this would be to use Amazon Athena to read and combine the files. Athena is a managed query service based on Presto that can read many different file formats.
The steps flow would be:
- Create a table definition in Athena that defines the input file formats and the location of the input data
- (You can use an AWS Glue crawler to do this for you)
- Use CREATE TABLE AS to query the source table
- This will retrieve data from the source files and write the output to a new location
- You can specify the output format and location
Think of Athena as a "query layer" on top of Amazon S3. It reads the input from all files in a given S3 directory and can then output the results back to S3. You can do a simple SELECT * to copy all the data, or you can choose to manipulate the results by selecting only desired fields and entries (using SELECT and WHERE).
Athena can be run from the management console, or triggered via a normal AWS SDK (such as Java).
The benefit of using Athena is that there is no need to download the source files and upload the result — this will all be done by Athena.
Athena is charged based on the amount of data read from disk. Compressed files reduce this cost.
Related