How to speed up the same calculation on sub-combinations of pandas DataFrame columns?

user_9606450 0
Viewed 34

I'm looking to apply the same function to multiple sub-combinations of a pandas DataFrame. Imagine the full DataFrame having 15 columns, and I want to draw from this full DataFrame a sub-frame containing 10 columns, I would have 3003 such sub-frames in total. My current approach is to use multiprocessing which works well for a full DataFrame with about 20 columns - 184,756 combinations, however the real full frame has 50 columns leading to more than 10 billions combinations, after which it will take too long. Is there any library that would be suitable for this type of calculation ? I have used dask before and it's incredibly powerful but dask is only suitable for calculation on a single DataFrame, not different ones.

Thanks.

dask pandas python
1 Answers

It's hard to answer this question without a MVCE. The best path forward depends on what you want to do with your 10 billion DataFrame combinations (write them to disk, train models, aggregations, etc.).

I'll provide some high level advice:

  • using a columnar file format like Parquet allows for column pruning (grabbing certain columns rather than all of them), which can be memory efficient
  • Persisting the entire DataFrame in memory with ddf.persist() may be a good way for you to handle this combinations problem so you're not constantly reloading it

Feel free to add more detail about the problem and I can add a more detailed solution.

user_11251590
Related