Find if value in nested sublist is greater than X within Pandas Column

user_9392446 0
Viewed 310

dataframe:

col1
[[0.43], [0.46], [1.0], [0.323]]
[[0.33], [0.66], [1.0], [0.3412]]
[[0.27], [0.42], [0.13], [0.332]]

I'm trying to create a column based on: if there's a number in the nested list of col11 >= .5 then "yes" else "no".

Result:

col1                                col2
[[0.43], [0.46], [1.0], [0.323]]    yes
[[0.33], [0.66], [1.0], [0.3412]]   yes
[[0.27], [0.42], [0.13], [0.332]]   no

It would also be good to have a column with the location of where the number was in nested list that was >= .5. for the above df, this would be col3: 3,3,N/A

Thinking something like this:

for i in df.col1:
    print(i)
    if j in i >= .05:
        print(i,"yes")
    else:
        print(i,"no")
pandas nested-lists python
3 Answers

You can do

sub_df = df[pd.DataFrame(df['col1'].apply(lambda x : sum(x,[])).tolist()).ge(0.5).any(1).values]
user_79645270

A simple apply should do

df['col2'] = df['col1'].apply(lambda x: max(e[0] for e in x) > 0.5)

Output

                                col1   col2
0   [[0.43], [0.46], [1.0], [0.323]]   True
1  [[0.33], [0.66], [1.0], [0.3412]]   True
2  [[0.27], [0.42], [0.13], [0.332]]  False
user_66928980

Use list comprehension with any. any is faster than max due it short-circuits on True

df['col2'] =  [any(y[0] >= 0.5 for y in x) for x in df.col1]

Out[1815]:
                                col1   col2
0   [[0.43], [0.46], [1.0], [0.323]]   True
1  [[0.33], [0.66], [1.0], [0.3412]]   True
2  [[0.27], [0.42], [0.13], [0.332]]  False

Some timings:

df = pd.concat([df]*10000, ignore_index=True)

In [1820]: %timeit  [any(y[0] >= 0.5 for y in x) for x in df.col1]
41.2 ms ± 1.03 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [1823]: %timeit df['col1'].apply(lambda x: max(e[0] for e in x) >= 0.5)
51.9 ms ± 1.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

side-test list comprehension with max

In [1827]: %timeit [max(y[0] for y in x) >= 0.5 for x in df.col1]
49.9 ms ± 2.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

So, any is faster although not significant.

user_101892140
Related