Failed to filter rows containing a specific value in the index column after resetting index

user_5336013 0
Viewed 901

I'm organizing data of a number of plans, which contains the information of the phase of the plan, P(Preliminary) or F(Final). I'm using the methods shown in the examples in the pandas documentation.

df1 = pd.read_excel('FilePath', sheetname = 'ForFilter')
df1


landuse_SUB_ID  TYPE    RECD_DATE   PHASE   LAND_USE    CPACTIONDA
0   24  1   2000-04-07  P   ROW 2000-05-04
1   24  1   2000-04-07  P   NONE    2000-05-04
2   25  1   2000-08-10  P   COMM    2000-09-08
3   34  1   2000-04-14  F   REC 2000-04-14
4   34  1   2000-04-14  F   SFD 2000-04-14
5   35  1   2000-01-20  P   NONE    2000-02-02
6   42  1   2000-04-04  P   SFD 2000-05-01
7   42  1   2000-12-06  P   SFD 2001-01-03
8   43  1   2000-09-07  P   NONE    2000-09-21
9   51  1   2000-11-10  P   NONE    2000-11-28
10  53  1   2000-02-22  F   SFD 2000-02-22

After playing with the methods in the example (using like and regex), it seems to me that these methods can only filter the values in the index column. Therefore I changed the index:

df1_filter1 = df1.set_index('PHASE')


landuse_SUB_ID  TYPE    RECD_DATE   LAND_USE    CPACTIONDA
PHASE                   
P   24  1   2000-04-07  ROW 2000-05-04
P   24  1   2000-04-07  NONE    2000-05-04
P   25  1   2000-08-10  COMM    2000-09-08
F   34  1   2000-04-14  REC 2000-04-14
F   34  1   2000-04-14  SFD 2000-04-14
P   35  1   2000-01-20  NONE    2000-02-02
P   42  1   2000-04-04  SFD 2000-05-01
P   42  1   2000-12-06  SFD 2001-01-03
P   43  1   2000-09-07  NONE    2000-09-21
P   51  1   2000-11-10  NONE    2000-11-28
F   53  1   2000-02-22  SFD 2000-02-22

Now the data frame is using Phase as index, I used the like method to filter df1_filter1:

df1_filter1.filter(like = 'F', axis = 0)

I get the error

"ValueError: cannot reindex from a duplicate axis"

This seems like a really simple operation to me, so I'm just wondering what I did wrong to have caused this error. And what shall be the best method (fewest steps and cleanest code) for my question.

pandas filter python
2 Answers

filter may intuitively feel like the right function, but you almost certainly should use loc to filter your data (on your examples link above, it says "See also: loc" in a big yellow box). For this simple example, you could also use boolean indexing:

>>> df1.loc[df1['PHASE'] == 'F']  # or boolean indexing via df1[df1['PHASE'] == 'F']
    landuse_SUB_ID  TYPE   RECD_DATE PHASE LAND_USE  CPACTIONDA
3               34     1  2000-04-14     F      REC  2000-04-14
4               34     1  2000-04-14     F      SFD  2000-04-14
10              53     1  2000-02-22     F      SFD  2000-02-22
user_24118020

As already pointed out, for this task, filter is not required. In @Alexander's answer loc is used. As an alternative, you could also use query:

df1.query('PHASE == "F"')

    landuse_SUB_ID  TYPE   RECD_DATE PHASE LAND_USE  CPACTIONDA
3               34     1  2000-04-14     F      REC  2000-04-14
4               34     1  2000-04-14     F      SFD  2000-04-14
10              53     1  2000-02-22     F      SFD  2000-02-22

filter works fine for me too:

df1_filter1 = df1.set_index('PHASE')
df1_filter1.filter(like='F', axis=0)

       landuse_SUB_ID  TYPE   RECD_DATE LAND_USE  CPACTIONDA
PHASE                                                       
F                  34     1  2000-04-14      REC  2000-04-14
F                  34     1  2000-04-14      SFD  2000-04-14
F                  53     1  2000-02-22      SFD  2000-02-22

Regarding your question about the use of regex:

df2 = df1.set_index('LAND_USE')
df2.filter(regex="E$", axis=0)

          landuse_SUB_ID  TYPE   RECD_DATE PHASE  CPACTIONDA
LAND_USE                                                    
NONE                  24     1  2000-04-07     P  2000-05-04
NONE                  35     1  2000-01-20     P  2000-02-02
NONE                  43     1  2000-09-07     P  2000-09-21
NONE                  51     1  2000-11-10     P  2000-11-28

Here you filter all rows that end with an E.

The same you could achieve by e.g. doing:

df1[df1['LAND_USE'].str.endswith('E')]

   landuse_SUB_ID  TYPE   RECD_DATE PHASE LAND_USE  CPACTIONDA
1              24     1  2000-04-07     P     NONE  2000-05-04
5              35     1  2000-01-20     P     NONE  2000-02-02
8              43     1  2000-09-07     P     NONE  2000-09-21
9              51     1  2000-11-10     P     NONE  2000-11-28

The problem with filter here is that you then create an index with non-unique values which is usually a bad idea. So, I would go with .loc or .query.

user_15340170
Related