Does numpy.argsort() increase (a lot) the size of an array?

LucianoSaldivia 0
Viewed 27

I have a numpy.array called combinations (2D array of 336^3=37,933,056 by 3 np.float32), and also an err_np (1D array of 336^3=37,933,056 np.float16), storing the %error of each 3 value combination.

I need to sort combinations by their absolute error, so i do:

from sys import getsizeof as size
import numpy as np

# Some stuff gets done here

print(size(combinations)) # 120 - **EITHER THIS IS TOO LOW**
print(len(combinations))  # 37,933,056
print(combinations)       # array OK

combinations = combinations[abs(err_np).argsort()] 

print(size(combinations)) # 455,196,792 - **OR THIS IS TOO HIGH**
print(len(combinations))  # 37,933,056
print(combinations)       # array sorted OK

I assumed the size before sorting was wrong, but i remind you size() is sys.getsizeof(), and the whole array is in there...

Is there a way to sort an np.array using less memory? It should also be as time-efficient as possible, since the size of the array. I currently del err_np after sorting combinations so i don't care about it.

numpy arrays python
1 Answers

So combinations is (37,933,056, 3) shape array. len() only displays the first dimension.

Then it's data takes up:

In [182]: 37933056*3*4
Out[182]: 455196672

bytes - that's the shape times 4 bytes per element.

From the first size I'd say combinations is a view of some other array. The 120 only records the base array size, not the shared data memory.

After the sort, combinations is a 'copy', with its own databuffer, and size reports the full size.

getsizeof is only useful if you understand what it's reporting. It's ok if the array is not a view, but that size can just as easily be calculated from shape and dtype. getsizeof is even less useful when looking at things like lists.

hpaulj0
Related