how to filter a dataframe using a feature list where column is a string description

Raul Santaella 0
Viewed 37

i tried looking at other responses and what i found didn't work.

i have a dataframe where a "description" column contains a long string. using bamboolib i get a proper result inputing the feature values manually.

filter_df = df.loc[(((df['description'].str.contains('estudio', case=False, regex=True, na=False)) | (df['description'].str.contains('family', case=False, regex=True, na=False))) | (df['description'].str.contains('family room', case=False, regex=True, na=False))) | (df['description'].str.contains('estar familiar', case=False, regex=True, na=False))]

how could a loop or other method replicate the same or similar filter using a list of the features.

ie:

feature_list = [ 'Estudio' , 'Family room' , 'Estar familiar' , 'Walking Closet' , 'Walk-in Closet' ,'Walk in closet' , 'Walkin closet' , 'Walkin clóset' , 'Walk in Clóset', 'Walk-in Clóset', 'Walking Clóset']

i would go with filtering with .loc[] and then prolly appending but that seems cumbersome and hopefully someone more experienced could share a streamlined way.

and even my attempts to filter using the following options returned empty dataframes, where they should not be empty in all cases

def filter_features(df,feature_list):
for idx, feature in enumerate(feature_list):
    print(feature)
    print(feature_list[idx])
    filter_df = df.loc[(df['description'].str.contains(feature, case=False, regex=True, na=False)) ]
    print(filter_df)
    
def filter_features(df,feature_list):
for idx, feature in enumerate(feature_list):
    print(feature)
    print(feature_list[idx])
    filter_df = df.loc[(df['description'].str.contains(feature_list[idx], case=False, regex=True, na=False)) ]
    print(filter_df)
    
def filter_features(df,feature_list):
for idx, feature in enumerate(feature_list):
    print(feature)
    print(feature_list[idx])
    filter_df = df.loc[(df['description'].str.contains(str(feature), case=False, regex=True, na=False)) ]
    print(filter_df)

====

this was one of the previous responses that i found that did not work for me

def filter_features(df,feature_list):
# Solution:
conditions = False
for f in feature_list:
    print(f)
    conditions = conditions | (df["description"].str.contains(f))
    print(conditions)
df.loc[conditions]
print(df)
string filter pandas python
1 Answers

Does this help in any way?

import random

feature_list = [ 'Estudio' , 'Family room' , 'Estar familiar' , 'Walking Closet' , 'Walk-in Closet' ,'Walk in closet' , 'Walkin closet' , 'Walkin clóset' , 'Walk in Clóset', 'Walk-in Clóset', 'Walking Clóset']
feature_list_incl_rubbish = feature_list + ['Jumping closet', 'Curling closet', 'Complaining closet']

df_unfiltered = pd.DataFrame(data = [random.choice(feature_list_incl_rubbish) for _ in range(0, 10000)], columns = ['description'])
print(f'Number of unique items in df_unfiltered: {df_unfiltered.description.nunique()}')

df_filtered = df_unfiltered.loc[lambda x: pd.DataFrame([x['description'].str.contains(y, case=False, regex=True, na=False) for y in feature_list]).any()]
print(f'Number of unique items in df_filtered  : {df_filtered.description.nunique()}')
abbasador0
Related