Basically, I have csv_events in my S3_bucket(s3://csv_events/user=111/year=2020/month=07/no.of.csv files). I want to convert these events into parquet format and want to store the results into another S3_bucket(s3://parquet_events/user=111/year=2020/month=07/parquet_files).
My Approach: First, i created a glue-crawler to crawl csv_events and created a athena_table(csv_events_table). Then Created a Glue-job, which will take csv_events_table as a input and convert those events into parquet and stored the results into S3. Finally, created another table for this parquet_events(parquet_events_table).
My approach is similar to this: https://www.powerupcloud.com/how-to-convert-historical-data-into-parquet-format-with-date-partitioning/
it is working fine, but i end up with having two athena_tables(csv_events_table,parquet_events_table).
Is there any way to directly access s3 data into glue job and convert it into parquet format? So that i will have only one athena_table(parquet_events_table)
Please let me know.
Regards -Siva