I have an incoming csv, with some unknown header. I read in some mappings from a db into the pipeline.
I need to rename the headers using the mappings and end up with a schema that allows for null/none values everywhere.
I read in the csv as a dataframe, which is great, and then I hit a wall - I could easily do a df.rename(columns=headers), but headers is a pcollection, and apparently DataFrameTransform doesn't play nice with side inputs.
Anyone know how the heck I can get these two things to play nicely? Thanks