Deleting colums of dataFrame where row value is constant for all rows

user_2543055 0
Viewed 73

Starting with a pandas DataFrame such as

import pandas as pd

df = pd.DataFrame(
    [[0, 3, 1.4, 3], [0, 3, 1.3, 1], [0, 3, 0.5, 3]]
)

or visually:

   0  1  2    3  
0[[0, 3, 1.4, 3] 
1 [0, 3, 1.3, 1]
1 [0, 3, 0.5, 3]]

and given a special value x_1=3

What would be a smart and scaling way to come up with a DataFrame that deletes all columns in df with a constant value x in EACH row?

The result in this example would be the dataFrame df without column 1.

df_altered =

   0  1    2 
0[[0, 1.4, 3] 
1 [0, 1.3, 1]
2 [0, 0.5, 3]]

In a small DataFrame I could itterate over all rows for each column but that would not scale and work with large DataFrames.

pandas dataframe python
4 Answers

You can use pd.drop():

df.drop(columns=df.columns[(df == 3).all()])

Output:

    0   2   3
0   0   1.4 3
1   0   1.3 1
2   0   0.5 3
user_133280100

One way is to determine the columns with equal values using:

>>> (df == df.iloc[0]).all(axis=0)
0     True
1     True
2    False
3    False
dtype: bool

Then extract the inverse of the above mask:

>>> df.iloc[:, ~(df == df.iloc[0]).all(axis=0).to_numpy()]
     2  3
0  1.4  3
1  1.3  1
2  0.5  3
user_3651020

You can try this:

## find the unique val in each column
no_unique_val = df.nunique()

val = 3
for column_name in no_unique_val.index:    
    if 1 == no_unique_val[column_name] and val == df[column_name].values[0]:            
        df.drop(column_name,axis=1, inplace=True)

output:

enter image description here

user_56470380

You can use df.ne and then df.any as boolean mask

df.loc[:, df.ne(3).any()]

   0    2  3
0  0  1.4  3
1  0  1.3  1
2  0  0.5  3
user_124164530
Related