When slicing a dataframe using loc,
df.loc[start:end]
both start and end are included. Is there an easy way to exclude the end when using loc?
When slicing a dataframe using loc,
df.loc[start:end]
both start and end are included. Is there an easy way to exclude the end when using loc?
None of the answers addresses the situation where end is not part of the index.
The more general solution is simply comparing the index to start and end, that way you can enforce either of them being inclusive of exclusive.
df[(df.index >= start) & (df.index < end)]
For instance:
>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame(
{
"x": np.arange(48),
"y": np.arange(48) * 2,
},
index=pd.date_range("2020-01-01 00:00:00", freq="1H", periods=48)
)
>>> start = "2020-01-01 14:00"
>>> end = "2020-01-01 19:30" # this is not in the index
>>> df[(df.index >= start) & (df.index < end)]
x y
2020-01-01 14:00:00 14 28
2020-01-01 15:00:00 15 30
2020-01-01 16:00:00 16 32
2020-01-01 17:00:00 17 34
2020-01-01 18:00:00 18 36
2020-01-01 19:00:00 19 38
For slicing a DatetimeIndex, you can try this. It will grab everything up to one nanosecond before your end time. This will exclude the end time (assuming you aren't using ns precision), but not necessarily the last time.
df.loc[start:(end - pd.Timedelta('1ns'))]
pd.RangeIndex can be used instead for slicing indices with .loc with an exclusive stop provided that the index has integer dtype. Here is a straightforward helper:
class _eidx:
def __getitem__(self, s: slice) -> pd.RangeIndex:
return pd.RangeIndex(s.start, s.stop, s.step)
eidx = _eidx()
Example:
df = pd.DataFrame({"x": range(10), "y": range(10, 20)})
print(df.loc[eidx[3:5]])
x y
3 3 13
4 4 14
An even simpler way is just using python range:
print(df.loc[range(3, 5)])
x y
3 3 13
4 4 14
There doesn't seem to be any really neat way to do this, but I would favour solutions which are expressive (is it clear what I'm trying to do?).
For this reason, I like this solution even though it's somewhat basic and a bit clumsy.
A more robust, expressive and, I think, performant version of this same idea would be to first create the inclusive slice, then filter the result to exclude the end-point:
df.loc[start:end][lambda _: _.index != end]
This solution is reasonably fast (I've set s = start; e = end) and done it with a Series called ts:
In [1]: %timeit ts[s:e]
135 µs ± 1.07 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [2]: %timeit ts[(ts.index >= s) & (ts.index < e)]
45.1 ms ± 142 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [3]: %timeit ts[s:e][lambda s: s.index != e]
299 µs ± 1.75 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
It can be made even more readable by allowing an intermediate variable:
inclusive = df.loc[start:end]
exclusive = inclusive[inclusive.index != end]