I have a DataArray which is too large to convert to a NumPy array:
da = xr.open_dataset('tprate.grib', engine='cfgrib').tprate
da.values
I got:
Unable to allocate 8.56 TiB for an array with shape (28, 7008, 185, 180, 360) and data type float32
If I isel part of the DataArray by the first two dimensions, it is able to convert:
ncep_da.isel(number=0, time=0).values
But if I isel the last two dimensions, the memory used remained unchanged:
ncep_da.isel(latitude=0, longitude=0).values
I still got:
Unable to allocate 8.56 TiB for an array with shape (28, 7008, 185, 180, 360) and data type float32
I guess this is because the second sub array is not stored sequentially on the disk as the rightmost dimension is the inner loop of the data.
Is there any way to extract the second sub array and convert it to NumPy array efficiently?