Creating new dataframe by appending rows from an old dataframe

user_13684209 0
Viewed 55

I'm trying to create a dataframe by selecting rows that meet only specific conditions from a different dataframe.

Technicians can only select one of several fields for Column 1 using a dropdown menu so I want to specify the specific field. However, column 2 is a freetext entry therefore I'm looking for two specific key words with any type of spelling/case.

I want all columns from the rows in the new dataframe.

Any help or insight would be much appreciated.

import pandas as pd

df = pd.read_excel (r'File.xlsx, sheet_name = 'Sheet1')

filter = ['x', 'y']
columns=df.columns

data = pd.DataFrame(columns=columns)
for row in df.iterrows():
    if 'Column 1' == 'a':
        row.data.append()
    elif df['Column 2'].str.contains('filter', case = 'false'):
        row.data.append()
        
print(data.head())
pandas dataframe python loops
1 Answers

In general, it's best to have a vectorized solution to things, so I'll put my solution as follows (there are many ways to do this, this is one of the ways that came to my head). Here, you can use a simple boolean mask to filter out some specific rows that you don't want, since you've already clearly defined your criteria (df['Column 1'] == 'a' or df['Column 2'].str.contains('filter', case = 'false')).

As such, you can simply create a boolean mask that includes this criteria. By itself, df['Column 1'] == 'a' will create an indexing dataframe with the structure of [1, 0, 1, 1, ...], where each number corresponds to whether it's true in the original array. Once you have that, you can simply index back into the original array with df[df['Column 1'] == 'a'] to return your filtered array.

Of course, since you have two conditions here (which seem to follow an "or" clause), you can simply feed both of these conditions into the boolean mask, such as df[df['Column 1'] == 'a' & df['Column 2'].str.contains('filter', case = 'false')].

I'm not at my development computer, so this might not work as expected due to a couple minor issues, but this is the general idea. This line should replace your entire df.iterrows block. Hope this helps :)

user_67948730
Related