pandas transform string to datetype. What happened in 1678 January the 1st?

user_7168098 0
Viewed 78

When transforming string data of a dataframe column into datetime I encountered the following that strikes me:

import pandas as pd
mydates = {'a':'2008-04-11','b':'2007-06-14','c':'2001-01-08','d':'2001-07-31'}
myDF = pd.DataFrame.from_dict(mydates,orient='index', columns=['dates'])

Now in order to transform the strings into datetype I proceed as follows:

myDF['as_date'] = pd.to_datetime(myDF['dates'].astype('str').str.replace('-',''), format='%Y%m%d')

all fine.

BUT... I started to change the date and wrote 'b':'1090-06-14' -> it does not work
but the end of WWII works: 'b':'1945-09-02'
and the start of the French revolution as well: 'b':'1789-05-05'

trying around I found that the limit is 1678-01-01, so if you do 'b':'1678-01-01' it works, but for any date earlier than that it does not work anymore.

Not that this is important programmatically (I don't think there is much to include in a data frame preceding that date) but WHY DOES THIS happen?

thanks

pandas dataframe date datetime python
1 Answers

Here the answer:
Since pandas represents timestamps in nanosecond resolution, the time span that can be represented using a 64-bit integer is limited to approximately 584 years:

In [92]: pd.Timestamp.min Out[92]: Timestamp('1677-09-21 00:12:43.145225')

In [93]: pd.Timestamp.max Out[93]: Timestamp('2262-04-11 23:47:16.854775807')

user_71680980
Related