I have two NumPy arrays, A and B.
A has shape (1000000, 512) and B has shape (1000000, 4) I am trying to concatenate them to get an array of shape (1000000, 516).
My problem is that np.concatenate([A, B], axis=1) takes way too much memory for some reason. Checking my memory usage, A and B take approximately 5GB (together), but when I run the concatenation my ram usage jumps by approximately 16GB.
Why is this happening? Is there a more efficient way of doing this operation for large arrays?
I tried doing the concatenation in batches but it doesn't help much and seems like the memory is not freed between iterations.