Given an input CSV (stored locally or in Google Cloud Storage) of
a,b,c
1,2,3
4,5,6
How can I get a PCollection with value
{a: 1, b: 2, c: 3}
{a: 4, b: 5, c: 6}
without knowing the name of the CSV headers in advance?
Convert csv to dict in Apache Beam without knowing header fieldnames
Viewed 674
1 Answers
There are two options here.
(1) You can use beam.io.ReadFromText skipping the header, followed by beam.Map(lambda line: zip(header_names, line.split(',')). This won't handle quoting, etc. (though could be adapted to do so, possibly using the csv module, though handling multi-line rows won'w work with this method).
(2) You can use the Beam dataframes API to do this, e.g.
from apache_beam.dataframe.io import read_csv
with beam.Pipeline as p:
df = p | beam.dataframe.io.read_csv("/path/to/filepattern")
# Here you can use df as if it were a Pandas dataframe,
# or you can convert it into a PCollection of dicts with
# pcoll = beam.dataframe.convert.to_pcollection(df)
Related