Why does pd.to_datetime use different default Y/M/D based on whether format is specified

user_4333359 0
Viewed 227

It's somewhat of a weird use case, but when converting a time-like value with pd.to_datetime pandas uses different values for the year, month, and day depending upon whether the format is specified.

pd.to_datetime('02:12:11', format='%H:%M:%S')
#Timestamp('1900-01-01 02:12:11')

pd.to_datetime('02:12:11')
#Timestamp('2021-03-17 02:12:11')

I had assumed that 1900-01-01T00:00:00.000 would be used to fill missing components in all cases, consistent with the datetime Technical Detail, (what the pandas docs link to) but somehow it decides to use 'today' to get the other components without a format.

Is this documented somewhere or expected behavior?

pandas python
2 Answers

If I understand the source code correctly, if no datetime.datetime object is passed to objects_to_datetime64ns then it defaults to datetime.now unless you specifiy the format.

I have to say going through the source code gives me a greater respect to the core maintainers of the project.

python3.8 > site-packages > dateutil > parser._parser.py

using pd.to_datetime('01:04:06') as an example.

if parserinfo:
    return parser(parserinfo).parse(timestr, **kwargs)
else:
    return DEFAULTPARSER.parse(timestr, **kwargs)

enter image description here

enter image description here

# the line of code in question. 
if default is None:
        default = datetime.datetime.now().replace(hour=0, minute=0,
                                                  second=0, microsecond=0)

This, from what i understand returns datetime.datetime(2021, 3, 17, 1, 4, 6) to

result, tz_parsed = tslib.array_to_datetime(
            data,
            errors=errors,
            utc=utc,
            dayfirst=dayfirst,
            yearfirst=yearfirst,
            require_iso8601=require_iso8601,
        )

Which returns your datetime object.

As if this is intended behavior I would urge on the side of yes as the code has been written in such a manner, however until it's made clear in the documents I would not give absolute certainty as pandas relies on the standard python lib for this conversion.

call stack for more information :

enter image description here

user_93751020

pandas depends on other libraries to do parsing, including python's datetime library and the dateutil package.

import pandas as pd
from datetime import datetime as dt
from dateutil import parser

print("pandas datetime with format ",pd.to_datetime('02:12:11', format='%H:%M:%S'))
print("datetime with format ", dt.strptime('02:12:11', '%H:%M:%S'))

print("pandas datetime without format ", pd.to_datetime('02:12:11'))

print("dateutil parser without format ", parser.parse('02:12:11'))

produces

pandas datetime with format  1900-01-01 02:12:11
datetime with format  1900-01-01 02:12:11
pandas datetime without format  2021-03-17 02:12:11
dateutil parser without format  2021-03-17 02:12:11

So the issue here is that the dateutil package uses the current date as the default, while strptime() uses the Unix origin.

user_19703540
Related