Convert pyarrow timezone aware timestamp to utc during parquet write

Viewed 28

I'm using pyarrow to read a csv and create timezone aware timestamps. And then i want to preserve that timezone information when writing to parquet. But parquet timestamps don't have timezone info.

Can i convert my timezone-aware timestamps to UTC somehow before writing to parquet?

This is my code, that loses timezone info when writing to parquet:

    schema = pa.schema(
        [
            ("Code", pa.string()),
            ("Timestamp", pa.timestamp("s", tz='America/Los_Angeles')),
            ("Value", pa.float64()),
            ("Status", pa.string()),
        ]
    )
    read_options=pa_csv.ReadOptions(column_names=schema.names),
    convert_options=pa_csv.ConvertOptions(
        timestamp_parsers=["%-m/%-d/%Y %-H:%M:%S %p"],
        column_types=schema,
    ),
    arrow = pa_csv.read_csv(
       fp,
       read_options=read_options,
       convert_options=convert_options,
    )
    pa_parquet.write_table(arrow, outfile)

0 Answers
Related