Why does a derived MultiIndex retain unused level data from the original index in pandas?

user_1028133 0
Viewed 37

When a filtered MultiIndex is derived from a larger MultiIndex instance, it appears that there's a discrepancy between the level values returned by MultiIndex.levels and MutliIndex.get_level_values():

import pandas as pd 
times = pd.date_range('2012-01-01', periods=365, freq='1d')
colors = ['red', 'blue']

index = pd.MultiIndex.from_product([times, colors])

new_index = index[index.get_level_values(0) > '20120701']

new_index.get_level_values(0) # includes only dates starting from 2012-07-02

results in:

DatetimeIndex(['2012-07-02', '2012-07-02', '2012-07-03', '2012-07-03',
           '2012-07-04', '2012-07-04', '2012-07-05', '2012-07-05',
           '2012-07-06', '2012-07-06',
           ...
           '2012-12-26', '2012-12-26', '2012-12-27', '2012-12-27',
           '2012-12-28', '2012-12-28', '2012-12-29', '2012-12-29',
           '2012-12-30', '2012-12-30'],
          dtype='datetime64[ns]', length=364, freq=None)

but levels[0] contains dates starting in Jan 2012:

new_index.levels[0] # includes all dates in the original index starting 2012-01-01

will yield:

DatetimeIndex(['2012-01-01', '2012-01-02', '2012-01-03', '2012-01-04',
           '2012-01-05', '2012-01-06', '2012-01-07', '2012-01-08',
           '2012-01-09', '2012-01-10',
           ...
           '2012-12-21', '2012-12-22', '2012-12-23', '2012-12-24',
           '2012-12-25', '2012-12-26', '2012-12-27', '2012-12-28',
           '2012-12-29', '2012-12-30'],
          dtype='datetime64[ns]', length=365, freq='D')

I would expect .get_level_values() to be consistent with .levels, so I'm wondering if there's a good reason for the derived index to retain the values of the original instance?

PS: as pointed out in jezrael's answer, MultiIndex.remove_unused_levels will remove the redundant dates

pandas
1 Answers

If use MultiIndex.remove_unused_levels then get:

new_index = index[index.get_level_values(0) > '20120701'].remove_unused_levels()

All value of first level by MultiIndex.get_level_values:

print (new_index.get_level_values(0))
DatetimeIndex(['2012-07-02', '2012-07-02', '2012-07-03', '2012-07-03',
               '2012-07-04', '2012-07-04', '2012-07-05', '2012-07-05',
               '2012-07-06', '2012-07-06',
               ...
               '2012-12-26', '2012-12-26', '2012-12-27', '2012-12-27',
               '2012-12-28', '2012-12-28', '2012-12-29', '2012-12-29',
               '2012-12-30', '2012-12-30'],
              dtype='datetime64[ns]', length=364, freq=None)

All unique values in first level:

print (new_index.levels[0])
DatetimeIndex(['2012-07-02', '2012-07-03', '2012-07-04', '2012-07-05',
               '2012-07-06', '2012-07-07', '2012-07-08', '2012-07-09',
               '2012-07-10', '2012-07-11',
               ...
               '2012-12-21', '2012-12-22', '2012-12-23', '2012-12-24',
               '2012-12-25', '2012-12-26', '2012-12-27', '2012-12-28',
               '2012-12-29', '2012-12-30'],
              dtype='datetime64[ns]', length=182, freq='D')
user_29010020
Related