List of Dataframes, drop Dataframe column (columns have different names) if row contains a special string

user_11196407 0
Viewed 232

What i have is a list of Dataframes.

What is important to note is that the shape of the dataframes differ between 2-7 columns, also the columns are named between 0 & len of the column (e.g. df1 has 5 columns named 0,1,2,3,4 etc. df2 has 4 columns named 0,1,2,3)

I would like is to check if a row in a column contains a certain string, then delete that column.

list_dfs1=[df1,df2,df3...df100]

What i have done so far is the below & i get an error that column 5 is not in axis (it is there for some DF)

for i, df in enumerate(list_dfs1):
    for index,row in df.iterrows():
        if np.where(row.str.contains("DEC")):
            df.drop(index, axis=1)

Any suggestions.

pandas python
3 Answers

You could try:

for df in list_dfs:
    for col in df.columns:
        # If you are unsure about column types, cast column as string:
        df[col] = df[col].astype(str)
        # Check if the column contains the string of interest
        if df[col].str.contains("DEC").any():
            df.drop(columns=[col], inplace=True)

If you know that all columns are of type string, you don't have to actually do df[col] = df[col].astype(str).

user_117640970

You can write a custom function that checks whether the dataframe has the pattern or not. You can use pd.Series.str.contains with pd.Series.any

def func(s):
    return s.str.contains('DEC').any()

list_df = [df.loc[:, ~df.apply(func)] for df in list_dfs1]
user_124164530

I would take another approach. I would concatenate the list into a data frame and then eliminate the column where finding the string

import pandas as pd
df = pd.concat(list_dfs1)

Let us say your condition was to eliminate any column with "DEC"

df.mask(df == "DEC").dropna(axis=1, how="any")
user_142380490
Related