I'm using Tree objects in python to manage my data for regression and inference at each tree node. The tree nodes often use outputs from other tree nodes as inputs.
I'm getting to the point where GPU/CUDA acceleration is necessary, especially with some of the permutations that I'm expecting with some of the next features I'm building.
When I look at the CuPy and CUDA tutorials, most examples are focused on how to use a single dataset or single CuPy data object. The cited metrics show the most benefits with larger array sizes: (https://cupy.dev/). I presume on the back end this means the libraries are optimized for a single transfer of the entire matrix object to the GPU memory, then performing operations on it. But I don't see many examples of CUDA acceleration with tree-based datasets where the tree objects hold the relevant data.
I'm concerned that the existing method I'm using, which is a tree of "small" data objects will be sub-optimal with how these CUDA libraries are used. I could refactor the tree nodes to all point at a single data structure that I know will work with CUDA... but that's a lot of work, and I'd have to do all the work before I know if it's actually any better. It's also based on my very limited understanding of how pure datasets vs objects containing datasets are handled in GPU acceleration libraries.
Does anyone have experience with tree-based data structures in CUDA? Does a large number of small datasets create issues for CUDA? Is it worth refactoring my tree nodes to point at a single data object in order to optimize GPU use?