I'm using pyarrow to read a csv and create timezone aware timestamps. And then i want to preserve that timezone information when writing to parquet. But parquet timestamps don't have timezone info.
Can i convert my timezone-aware timestamps to UTC somehow before writing to parquet?
This is my code, that loses timezone info when writing to parquet:
schema = pa.schema(
[
("Code", pa.string()),
("Timestamp", pa.timestamp("s", tz='America/Los_Angeles')),
("Value", pa.float64()),
("Status", pa.string()),
]
)
read_options=pa_csv.ReadOptions(column_names=schema.names),
convert_options=pa_csv.ConvertOptions(
timestamp_parsers=["%-m/%-d/%Y %-H:%M:%S %p"],
column_types=schema,
),
arrow = pa_csv.read_csv(
fp,
read_options=read_options,
convert_options=convert_options,
)
pa_parquet.write_table(arrow, outfile)