I have so much of data in my local Postgres database on my machine. I need to denormalize the data present in this local database and get a query set in specific format which can be loaded directly into Redshift tables directly using Python.
I do have queries that I can run on the local database and get the query set in specific format that needs to be loaded to Redshift directly.
But there is so much data that I need to move from local to Redshift. For now, the only better way I could think of is exporting the queryset that I got into a .csv file which will be uploaded to an S3 bucket which will be directly copied into the Redshift tables using Python.
I'm just wondering if there is any alternative way to do this. something like streaming directly from Postgres database to AWS Redshift
Please let me know if the uploading and dumping the .csv is a better way or is there any other efficient way to achieve this.