So I'm loading up some files using np.load() in the size range 150MB to 250MB. each file contains an array that has 5 sub arrays with some data. some files are loaded in under a second while others take up to 5 seconds to load and since I have a lot of these files they take quite a while to work with them all due ot the slow loading times. But, I have noticed that if I split the files into 5 smaller files (1 sub array per file) then the loading time is always easily under a second per 5 files.
what could be causing this? how can I speed up np.load() without splitting each file into smaller files?