I have a very big numpy matrix. Luckily there are only 4 possible options for values in the matrix, let say 0,1,2,3. Now of course instead of using float32, I can set the type to int8, to save memory (and storage, later when I save it).
My values range is exactly 2 in the power of 2, so theoretically, I can use just 2 bits for each cell, and save about 4 times memory. Is it possible to give a specific number of bits for the dtype?
Ideally I would like to do something like:
compressed_matrix = matrix.astype(np.int2)
I know theoretically I can stuck few cells together, like:
for i in range(0,len(arr),4):
comp_arr[i/4] = arr[i+3] + 4 * arr[i+2] + 16 * arr[i+1] + 64 * arr[i]
And later it will be possible to reconstruct it. But in the compressed form it will be very hard to apply operations on this array, such as transpose it, sum along axis, etc.. This solution might be good only for saving the matrix in the disk.
EDIT:
I add here some code example of usage. Let say window0 and window2 are 2 matrix that satisfy this condition. Both very big, with values in range 0 to 3, and both in the same shape).
We have as well vectors name ref_freq and non_ref_freq, which are floats vector in the range between 0 to 1, and are in the same len, defined by ref_freq = 1 - non_ref_freq.
My current code, which works, but not on very large matrix is:
first_element = (ref_freq * window0) @ window0.T
second_element = (non_ref_freq * (2 - window2)) @ (2 - window2).T
similarity = (first_element + second_element) / 4
np.fill_diagonal(similarity, 0)
Any idea that can help me?
Thanks,
Shahar