how low in dataframe to find all elements of list

Viewed 151

I have a list:

elements = ['a', 'b', 'c', 'd']

And a dataframe that has some or all of the elements of my list:

       mycol
0      a
1      x
2      y
3      e
4      b
5      c
6      o
7      l
8      s
9      d
10     g

I want to know how low I have to search on my df to find all of the elements of my list. In this case the answer would be 10 because it is until where I found all the elements of my list.

Thanks

4 Answers

It's worth considering Barmar's comment. I couldn't get the fancier indexing answers to work with some bigger testing data, but Barmar's loop should be reliable:

Just loop over the dataframe indexes. If the current df element is in the list, remove it from the list. When the list becomes empty, the current index is the answer.

def idxall(series, elements):
    for i, e in enumerate(series.to_numpy()): # faster than series.items()
        if e in elements:
            elements.remove(e)
            if not elements:
                return i + 1
    return np.nan

Timings

Given df = pd.DataFrame({'mycol': np.random.choice(list(string.ascii_lowercase), size=1000)}):

%timeit tdy_idxall(df.mycol, list(string.ascii_lowercase))
# 21.4 µs ± 7.44 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit henry_ecker_np_unique(df.mycol, list(string.ascii_lowercase))
# 379 µs ± 48.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit u12_forward_idxmax(df.mycol, list(string.ascii_lowercase)
# 538 µs ± 61.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit corralien_idxall(df.mycol, list(string.ascii_lowercase))
# 1.28 ms ± 243 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Verification

  • Using OP's sample:

    df = pd.DataFrame({'mycol': list('axyebcolsdg')})
    elements = list('abcd')
    
    idxall(df.mycol, elements)
    # 10
    
  • Using Henry's sample #1 (mixed order and duplicates):

    df = pd.DataFrame({'mycol': list('dxcabcodsdg')})
    elements = list('abcd')
    
    idxall(df.mycol, elements)
    # 5
    
  • Using Henry's sample #2 (not all elements found):

    df = pd.DataFrame({'mycol': list('dxcabcodsdg')})
    elements = list('abcz')
    
    idxall(df.mycol, elements)
    # nan
    

Try idxmax:

>>> df['mycol'].isin(elements)[::-1].idxmax()
9
>>> 

Edit:

For specifying all values in elements are in the dataframe, try:

x = df['mycol'].drop_duplicates().isin(elements).cumsum().eq(len(elements))
if x.any():
    print(x.idxmax())
else:
    print("Not all values are in the dataframe")

For your current dataframe:

9

For your dataframes where not all values are in the dataframe:

Not all values are in the dataframe

We can use np.unique with return_index=True in order to find the first instance of each unique value:

import numpy as np
import pandas as pd

elements = ['a', 'b', 'c', 'd']
df = pd.DataFrame({
    'mycol': ['a', 'x', 'y', 'e', 'b', 'c', 'o', 'l', 's', 'd', 'g']
})

# Find the first location where each unique value is found
a, b = np.unique(df['mycol'], return_index=True)
# Compare unique values to values we're looking for
m = (a == np.array(elements)[:, None])
# If we have a location for all elements
if m.any(axis=1).all():
    # Find the highest index value
    max_index = b[m.any(axis=0)].max()
    # Offset index by one to match expected output
    print('All values found by', max_index + 1)
else:
    # We couldn't find all elements
    print('Not all elements found.')
All values found by 10

Example with mixed order and duplicates:

elements = ['a', 'b', 'c', 'd']
df = pd.DataFrame({
    'mycol': ['d', 'x', 'c', 'a', 'b', 'c', 'o', 'd', 's', 'd', 'g']
})
   mycol
0      d
1      x
2      c
3      a
4      b
5      c
6      o
7      d
8      s
9      d
10     g
All values found by 5

Example with not all elements found:

elements = ['a', 'b', 'c', 'z']
df = pd.DataFrame({
    'mycol': ['d', 'x', 'c', 'a', 'b', 'c', 'o', 'd', 's', 'd', 'g']
})
   mycol
0      d
1      x
2      c
3      a
4      b
5      c
6      o
7      d
8      s
9      d
10     g
Not all elements found.  # (No z)

You can use pd.CategoricalDtype and use set to check if all elements are in the filtered dataframe:

def idxall(series, elements):
    out = series.astype(pd.CategoricalDtype(elements)) \
                .reset_index(drop=True) \
                .dropna().drop_duplicates()
    return out.index.max()+1 if not set(elements).difference(out) else np.nan
  1. Your sample:
df = pd.DataFrame({'mycol': list('axyebcolsdg')})
elements = list('abcd')
    
>>> idxall(df['mycol'], elements)
10
  1. Henry's sample #1 (mixed order and duplicates):
df = pd.DataFrame({'mycol': list('dxcabcodsdg')})
elements = list('abcd')

>>> idxall(df['mycol'], elements)
5
  1. Henry's sample #2 (not all elements found):
df = pd.DataFrame({'mycol': list('dxcabcodsdg')})
elements = list('abcz')

>>> idxall(df['mycol'], elements)
nan
Related