I have large datasets extracted from binary files. They consist in X, Y arrays containing positive decimal coordinates, and other arrays containing values associated to the XY coordinates (Z). I am interested in plotting 2D heatmaps. I have millions of X,Y,Z coordinates in each dataset, so I would like to resample them before generating images.
I could take 1 every Nth data chunk when reading the binary file, but this can affect the result to some extent, as some data are sampled closer than others.
I have tried filling an array and plotting it with imshow(), but it can take a lot of memory, even if most the array is empty. It is also highly limited in size...
I think the most accurate way would be to chose a radius or cell size, and combine/aggregate all the Z data located in each cell (by mean or average), and repeat that to cover the whole X and Y coordinates. However, I don't really see how to do this without calculating the distances between every (X,Y) coordinate set, which can take a lot of time. Are there any ways to perform this kind of operation fast?
I hope I was clear enough!