Including column headers in CSV export from RDS postgres db using AWS Data Pipeline?

user_8465623 0
Viewed 715

I set up an AWS data pipeline that dumps data from our RDS postgres db to a S3 location but the column headers are not included. I would like to avoid using the solution outlined here because it seems brittle (if the columns change, also want to avoid casting columns as string). Thanks in advance

ed. Here is the data pipeline configuration: diagram

amazon-web-services csv postgresql
2 Answers

I couldn't find a solution for this either, but a possible workaround is to use a temporary destination folder for data-pipeline and use a Lambda function to take those dumps, add the headers, and move them to the final dest. The best of this approach is that the column names are not hard-coded, so if the table changes the CSV will change too.

user_23547850
Related