I currently have a 400 GB postgresql database in rds that I would like to load in s3 in Parquet Format so as to use Athena and then quicksight on it.
I would like the data to be updated every week at least.
I tried to use Glue as suggested by this answer
It works perfectly with small tables.
However with a 300 GB table the connection fails after 2h. (I remember reading somewhere that Glue didn't manage well big postgresql datasets) Indeed Glue can't filter last week results, it has to load the entire results first.
Exporting snapshots from rds directly to s3 in parquet format worked perfectly for the whole dataset but I fear exporting 400GB of data every week could cost more than setup a way to do it incrementally.
Is this video an efficient way to do it ? Would someone see another way to do it ? Maybe with lambda and boto3? (but I fear size limitations in lambda functions) or Kinesis ? SQS ?
Thank you in advance