Dask: subset (or drop) rows from Dataframe by index

user_2328684 0
Viewed 1366

I'd like to take a subset of rows of a Dask dataframe based on a set of index keys. (Specifically, I want to find rows of ddf1 whose index is not in the index of ddf2.)

Both cache.drop([overlap_list]) and diff = cache[should_keep_bool_array] either throw a NotImplementedException or otherwise don't work.

What is the best way to do this?

dask python
2 Answers

I'm not sure this is the "best" way, but here's how I ended up doing it:

  1. Create a Pandas DataFrame with the index be the series of index keys I want to keep (e.g., pd.DataFrame(index=overlap_list))
  2. Inner join the Dask Dataframe
user_23286840
Related