Filtering if list elements are same in a pandas column

user_5852692 0
Viewed 222

I have a dataframe, which simply looks like this:

edges
      id       node_id     ...
0   'AX'    ['A', 'B']
1   'BX'    ['B', 'C']
2   'CX'    ['C', 'C']

The 'id' column has string elements, and the 'node_id' column has lists (with strings inside).

I would like to remove some elements from this pandas df, if their 'node_id' has 2 same string.

In above dataframe this would be the 2nd element since its 'node_id' has 'C' and 'C'.

To do that, I am using the following:

edges[edges['node_id'].apply(lambda x: x[0]) != edges['node_id'].apply(lambda x: x[1])]

However, since .apply() is not really time efficient, I am looking for a built-in pandas function. Is there any to achieve what I do?

pandas apply dataframe filter python
2 Answers

Try this:

df = pd.DataFrame({
    'id': ['AX', 'BX', 'CX'],
    'node_id': [['A','B'],['B', 'C'],['C', 'C']]})

mask = df['node_id'].apply(lambda x : x[0]!=x[1])

df = df[mask]

Output:

    id  node_id
0   AX  [A, B]
1   BX  [B, C]

If you have multi item in list you can try this:

df = pd.DataFrame({
    'id': ['AX', 'BX', 'CX'],
    'node_id': [['A', 'B', 'C'],['B', 'C', 'C'],['C', 'C', 'C']]})

mask = df['node_id'].apply(lambda x : len(x)==len(set(x)))

df = df[mask]

Output:

id  node_id
0   AX  [A, B, C]
user_17405770

Another version using only vectorized functions and without using .apply():

Filter to keep only rows with "length of unique items in the list" is equal to "length of the list":

Set mask for the condition "length of unique items in the list" == "length of the list", and then filter by the mask:

.map() each list in node_id with pd.unique to get the unique item list:

mask = df['node_id'].map(pd.unique).str.len() == df['node_id'].str.len()
df.loc[mask]

Data Input:

df = pd.DataFrame({
    'id':      ['AX',       'BX',         'CX'],
    'node_id': [['A','B'],  ['B', 'C'],   ['C', 'C']]})


   id node_id
0  AX  [A, B]
1  BX  [B, C]
2  CX  [C, C]

Result

   id node_id
0  AX  [A, B]
1  BX  [B, C]

Result:

user_150706970
Related