pandas: Locate/Select records where a column has list of nan

user_9996043 0
Viewed 106

I have a dataset where few of my columns have lists of float as their value for a record. I'm trying to locate/select all those records where say in my dataframe df for column x1 the records have [nan, nan]

        x1            x2
0 [21.25, 25.45]   string1
1 [45.25, 85.45]   string2 
2   [nan, nan]     string3
3 [96.25, 79.45]   string4
4 [12.25, 65.45]   string5
5   [nan, nan]     string6
6   [nan, nan]     string7

I want the result:

        x1            x2
2   [nan, nan]     string3
5   [nan, nan]     string6
6   [nan, nan]     string7

I know the way to do it if there are no lists just float values for each record, df.loc[df['x1'].isnull()]

But I can't seem to figure out how to do it if it's a list.

pandas dataframe python
3 Answers

Another method:

print(df[df['x1'].apply(lambda x: np.isnan(x).all())])

Prints:

           x1       x2
2  [nan, nan]  string3
5  [nan, nan]  string6
6  [nan, nan]  string7
user_100359850

You could work with numpy here to check where all values in a row are np.nan in one go.

Note that this assumes all values in x1 have the same amount of values.

df[np.isnan(np.stack(df.x1.to_numpy()).astype(float)).all(1)]

           x1       x2
2  [nan, nan]  string3
5  [nan, nan]  string6
6  [nan, nan]  string7
user_96986840

Let's try explode:

df[df['x1'].explode().isna().all(level=0)]

Output:

           x1       x2
2  [nan, nan]  string3
5  [nan, nan]  string6
6  [nan, nan]  string7
user_42384080
Related