How do I make Koalas preserve timezone information in a timezone-aware pandas Timestamp?
>>> import pandas as pd
>>> pds = pd.Series([pd.Timestamp('2018-11-06T14:00:00+00:00')])
>>> pds
0 2018-11-06 14:00:00+00:00
dtype: datetime64[ns, UTC]
See how it includes the +00:00 on the end, and dtype is datetime64[ns, UTC]. This is how I want it.
Now I convert it to koalas:
>>> import databricks.koalas as ks
>>> kss = ks.from_pandas(pds)
kss
0 2018-11-06 08:00:00
dtype: datetime64[ns]
It has lost the timezone information, and has subtracted 6 hours. Not sure why, since I am in US CDT, you would think it would ADD 5 hours rather than SUBCTRACT 6. And furthermore, now when I write it to CSV:
>>> kss.to_csv('output.csv')
It subtracts another six hours by appending -06:00:
2018-11-06T08:00:00.000-06:00
So after going from pandas -> koalas -> CSV my timestamp has lost 12 full hours: what started out as 2018-11-06 14:00:00+00:00 has become 2018-11-06T08:00:00.000-06:00