Create a new dataframe with the lines that were not included on other dataframe

user_12793293 0
Viewed 48

I need to create a new dataframe with the lines that were not included in the dataframe ''results'' but are on the dataframe ''df'' without creating a new filter.

I have no idea how to do it. Can you help me out? (sorry if noob question)

import pandas as pd

#Creating dataframe
d = {'col1': [1, 2,3,4,5,6,7,8,9,10], 'col2': ['a','b','b','b','c','d','c','a','z','c']}
df = pd.DataFrame(data=d)

#Finding the lines that contain a certain letter
a = df[df['col2'].str.contains("a")]
b = df[df['col2'].str.contains("b")]
c = df[df['col2'].str.contains("c")]

#Merge the 3 data frames
frames = [a, b, c]
results = pd.concat(frames)
print(results)
pandas dataframe python
3 Answers

Better solution should be create mask by all 3 masks in one with | for OR and then for non match rows invert mask by ~:

m = df['col2'].str.contains("a|b|c")
results = df[m]
print(results)
   col1 col2
0     1    a
1     2    b
2     3    b
3     4    b
4     5    c
6     7    c
7     8    a
9    10    c

df1 = df[~m]
print (df1)
   col1 col2
5     6    d
8     9    z

Your solution should be changed by filter non matched index values:

df1 = df[~df.index.isin(results.index)]
print (df1)
   col1 col2
5     6    d
8     9    z
user_29010020

First concat both df and results

new_df = pd.concat([df, results])

then drop all duplicates with keep=False parameter to remove all common entries from both dataframe, and you will get your required dataframe

new_df = new_df.drop_duplicates(keep=False)
user_110046760

Use df.index.difference - Will return the index that are not in other dataframe.

>>> df.iloc[df.index.difference(results.index)]
   col1 col2
5     6    d
8     9    z

Or

>>> df[~df.isin(results).all(1)]
   col1 col2
5     6    d
8     9    z
user_60756990
Related