Pandas read_csv() parses dates fine but can't index by date

Viewed 43

This is strange.

Data (csv):

Date,    Hr 1,Hr 2,Hr 3,..
20070701,1128,1072,1173,..
20070702,1131,1092,1287,..

Pretty vanilla use of pd.read_csv():

df = pd.read_csv(   filename,
                    parse_dates=['Date'],
                    index_col=['Date'])

Date seems to parse fine into the index:

print(df.index[:2])

Output:

DatetimeIndex(['2007-07-01', '2007-07-02'], dtype='datetime64[ns]', name='Date', freq=None)

Now if I try to index a single day?

print(df['2007-7-1']) # or any variation on "2007-07-01" etc

Output:

Traceback (most recent call last):
  File "/Users/mjw/opt/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '2007-7-1'
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "my_file.py", line 108, in <module>
    print(df['2007-7-1'])
  File "/Users/mjw/opt/anaconda3/lib/python3.8/site-packages/pandas/core/frame.py", line 2800, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/Users/mjw/opt/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2648, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '2007-7-1'

I've also tried to make sure the DatetimeIndex freq is set right

df = df.asfreq('d')

And I get the same junk.

But indexing by year and month works fine, or indexing by year-month-day after selecting a column:

print(df['2007-7']) # works
print(df['Hr 1']['2007-7-1']) # works

But this does not:

print(df['2007-7-1']['Hr 1'])

I can make a custom date parser but the point is that I shouldn't have to do that. "yyyymmdd" isn't exactly hard or unusual. Come on pandas.

Please and thank you!

1 Answers

Use .loc:

print(df.loc["2007-07-01"])

Prints:

    Hr 1    1128
Hr 2        1072
Hr 3        1173
Name: 2007-07-01 00:00:00, dtype: int64

For just value of "Hr 2" column:

print(df.loc["2007-07-01", "Hr 2"])

Prints:

1072
Related