When I attempt to write a parquet file utilizing a pandas.Timestamp as the index, I cannot read it back in. I get an error regarding timestamp conversion.
pytz.exceptions.UnknownTimeZoneError: '-5:00'
I have run this code against pyarrow 0.11.1 and it works fine, when switching to 0.15.0 it breaks. I think it's how pyarrow is treating datetime objects, parsing the timezone through pytz instead.
This code works for pyarrow 0.11.1 and breaks for 0.15.0
import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa
import json
myjson = ['{"received_time":"2019-10-13T00:00:09.915-05:00","mydata":1.0}',
'{"received_time":"2019-10-13T00:00:10.915-05:00","mydata":1.0}',
'{"received_time":"2019-10-13T00:00:11.915-05:00","mydata":1.0}']
iterator = 0
q=[]
for i in myjson:
q.append(json.loads(i))
iterator = iterator + 1
df = pd.DataFrame.from_records(q)
tindex = pd.to_datetime(df['received_time'],format='%Y-%m-%dT%H:%M:%S.%f%z')
df['received_time'] = tindex
df.set_index('received_time',inplace=True)
tbl = pa.Table.from_pandas(df)
pq.write_table(tbl,'tmp.pq')
readdf = pd.read_parquet('tmp.pq')
I expect this to work without error, instead I get this stacktrace:
Traceback (most recent call last):
File "<input>", line 21, in <module>
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/io/parquet.py", line 296, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/io/parquet.py", line 125, in read
path, columns=columns, **kwargs
File "pyarrow/array.pxi", line 468, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 1238, in pyarrow.lib.Table._to_pandas
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 697, in table_to_blockmanager
all_columns)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 778, in _reconstruct_index
table, result_table, descr, field_name_to_metadata)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 835, in _extract_index_level
.dt.tz_convert(col.type.tz))
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/accessor.py", line 93, in f
return self._delegate_method(name, *args, **kwargs)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/indexes/accessors.py", line 109, in _delegate_method
result = method(*args, **kwargs)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/accessor.py", line 93, in f
return self._delegate_method(name, *args, **kwargs)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/indexes/datetimelike.py", line 813, in _delegate_method
result = operator.methodcaller(name, *args, **kwargs)(self._data)
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 955, in tz_convert
tz = timezones.maybe_get_tz(tz)
File "pandas/_libs/tslibs/timezones.pyx", line 84, in pandas._libs.tslibs.timezones.maybe_get_tz
File "pandas/_libs/tslibs/timezones.pyx", line 99, in pandas._libs.tslibs.timezones.maybe_get_tz
File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pytz/__init__.py", line 181, in timezone
raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: '-05:00'
Here is the output of pandas.show_versions()
pd.show_versions()
INSTALLED VERSIONS
------------------
commit : None
python : 3.6.6.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-862.el7.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 0.25.1
numpy : 1.15.4
pytz : 2019.3
dateutil : 2.7.5
pip : 10.0.1
setuptools : 39.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.0.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.0
pytables : None
s3fs : None
scipy : 1.3.0
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
I'm not sure if this is a pyarrow issue or a pandas one, but I do know that pyarrow switched to processing datetimes like this:
if isinstance(col.type, pa.lib.TimestampType):
index_level = (pd.Series(values).dt.tz_localize('utc')
.dt.tz_convert(col.type.tz))
which calls into the pytz library, whereas before it didn't. Either I'm constructing the parquet file incorrectly (in which case there should be a parsing error on write) or there is something up with either how the pandas or parquet libraries handle timezones.
Any thoughts?