My dataset depends on a 3GB tensor. This tensor could either be on the CPU or the GPU. The bottleneck of my code is the data loading preprocessing. But I can't add more than a few workers without killing my RAM.
This sounds silly for me: why could each worker receives a copy of the 3GB tensor, when this one is exactly the same across each worker?
Is there any solution for letting the workers access to a single version of this tensor?
Thanks,