Pandas read_parquet pytz.exceptions.UnknownTimeZoneError

user_3723311 0
Viewed 515

I want to read a parquet folder contenant part of data with pandas.

I have datetime date as UTC in it and when I try to read it I got :

pytz.exceptions.UnknownTimeZoneError: '+00:00'

Cannot find out what is the issue.

parquet pyarrow pandas
1 Answers

I ran into this issue when populating a DataFrame with tz-aware timestamps that I got from a database via psycopg2. The issue was that Pandas Parquet code did not handle the timezone object from psycopg2 very well. Converting the data to pytz UTC solved the issue.

Here's a demo:

import datetime as dt
import pandas as pd
from psycopg2.tz import FixedOffsetTimezone

# Simulate getting a timestamp from the DB
psycopg2_utc = FixedOffsetTimezone()
timestamp = dt.datetime.now(psycopg2_utc)

df = pd.DataFrame({'foo': [timestamp]})

df.to_parquet('test1.parquet')
try:
    pd.read_parquet('test1.parquet')
except Exception as e:
    print(f'First load failed: {e}')

# Fix by converting to pytz UTC
df['foo'] = df['foo'].dt.tz_convert('UTC')

df.to_parquet('test2.parquet')
pd.read_parquet('test2.parquet')
print('Second load succeeded')

I've filed an issue at Pandas.

user_8573900
Related