How do I find where a Series/DataFrame containing intervals overlaps a given interval

Viewed 262

I have a DataFrame with price ranges for some items:

   price_low  price_high item
0         10          20    a
1          1           7    b
2         10          12    c
3         20          25    d
4          4           8    e
5          5          30    f
6         16          26    g

How do I figure out which items overlap a given price range, say $8 - $16?

Expected output:

   price_low  price_high item
0         10          20    a
2         10          12    c
4          4           8    e
5          5          30    f
6         16          26    g
2 Answers

As of pandas 0.24.0, you can use the IntervalArray.overlaps method, or alternatively the IntervalIndex.overlaps method:

# construct the IntervalArray
price_ivs = pd.arrays.IntervalArray.from_arrays(df['price_low'], df['price_high'], closed='both')

# define desired price Interval and use the overlaps method to restrict df
my_price = pd.Interval(8, 16, closed='both')
df = df[price_ivs.overlaps(my_price)]

For an interactive demonstration, first construct the example data:

In [1]: import pandas as pd; pd.__version__
Out[1]: '0.24.0rc1'

In [2]: df = pd.DataFrame({
   ...:     'price_low': [10, 1, 10, 20, 4, 5, 16],
   ...:     'price_high': [20, 7, 12, 25, 8, 30, 26],
   ...:     'item': list('abcdefg')
   ...: })

In [3]: df
Out[3]: 
   price_low  price_high item
0         10          20    a
1          1           7    b
2         10          12    c
3         20          25    d
4          4           8    e
5          5          30    f
6         16          26    g

Construct an IntervalArray from the DataFrame:

In [4]: price_ivs = pd.arrays.IntervalArray.from_arrays(
   ...:     df['price_low'], df['price_high'], closed='both')

In [5]: price_ivs
Out[5]: 
IntervalArray([[10, 20], [1, 7], [10, 12], [20, 25], [4, 8], [5, 30], [16, 26]],
              closed='both',
              dtype='interval[int64]')

Define the desired price Interval and use the overlaps method to get a boolean indexer:

In [6]: my_price = pd.Interval(8, 16, closed='both')

In [7]: idxr = price_ivs.overlaps(my_price)

In [8]: idxr
Out[8]: array([ True, False,  True, False,  True,  True,  True])

In [9]: df[idxr]
Out[9]: 
   price_low  price_high item
0         10          20    a
2         10          12    c
4          4           8    e
5          5          30    f
6         16          26    g

If you already have a column of prices as intervals (or a Series of intervals), you can use the array attribute to access the underlying IntervalArray and use the same method as above:

In [10]: df = pd.DataFrame({'price_ivs': price_ivs, 'item': list('abcdefg')})

In [11]: df
Out[11]: 
  price_ivs item
0  [10, 20]    a
1    [1, 7]    b
2  [10, 12]    c
3  [20, 25]    d
4    [4, 8]    e
5   [5, 30]    f
6  [16, 26]    g

In [12]: idxr = df['price_ivs'].array.overlaps(my_price)

In [13]: idxr
Out[13]: array([ True, False,  True, False,  True,  True,  True])

In [14]: df[idxr]
Out[14]: 
  price_ivs item
0  [10, 20]    a
2  [10, 12]    c
4    [4, 8]    e
5   [5, 30]    f
6  [16, 26]    g

root's answer is general and informative. In case you are looking for a quick solution for simple cases, you could try below

df[(df['price_low']<=16)&(df['price_high']>=8)]

It's a precondition for two intervals to overlap. It's easier to understand in this way: the lower boundary of the price range (8) should not be higher than price_high and its upper boundary (16) should not be lower than price_low.

Related