How to insert an array into another one using slices

user_6659081 0
Viewed 277

I want to create a large NumPy array (L) to hold the result of some operations. However, I can only compute one part of L at a time. Then, to have L, I need to create an array of zeros of shape L.shape, and fill it using these parts or subarrays. I'm currently able to do it, but in a very inefficient way.

If the shape of L is (x, y, z, a, b, c), then I'm creating a NumPy arrays of shape (x, y, z, 1, b, c) which correspond to the different parts of L, from part 0 to part a-1. I'm forced to create arrays of this particular shape due to the operations involved.

In order to fill the array of zeros, I'm creating one Pandas DataFrame per subarray (or part). Each dataframe contains the indices and the values of one subarray of shape (x, y, z, 1, b, c), like this:

index0 | index1 | index2 | index3 | index4 | index5 | value
------------------------------------------------------------
     0 |      0 |      0 |      0 |      0 |      0 | 434.2
     0 |      0 |      0 |      0 |      0 |      1 | 234.5

..., and so on.

Because of the shape (x, y, z, 1, b, c), index3 can only contain zeros. So, there's a change to make before the values can be inserted at the right index of L: the column at index3 will contain only 0s for the first subarray, only 1s for the second subarray, etc. So, df['index3'] = subarray_number, where subarray_number goes from 0 to a-1. Only the column at index3 is changed.

So, the fifth subarray represented as a dataframe would look like this:

index0 | index1 | index2 | index3 | index4 | index5 | value
------------------------------------------------------------
     0 |      0 |      0 |      4 |      0 |      0 | 434.2
     0 |      0 |      0 |      4 |      0 |      1 | 234.5
...
   x-1 |    y-1 |    z-1 |      4 |    b-1 |    c-1 | 371.8

After this, I only have to iterate over the rows of each of the dataframes using iterrows, and assign the values to the corresponding indices of the array of zeros, like this:

for subarray_df in subarrays_dfs:
    for i, row in subarray_df.iterrows():
        index0, index1, index2, index3, index4, index5, value = row
        L[index0][index1][index2][index3][index4][index5] = value

The problem is that converting the arrays to dataframes and then assigning the values one by one is expensive, especially for large arrays. I would like to insert the subarrays in L directly without having to go through this intermediate step.

I tried using slices but the generated array is not the one I expect. This is what I'm doing:

L[:subarray.shape[0], :subarray.shape[1], :subarray.shape[2], 
  subarray_number, :subarray.shape[4], :subarray.shape[5]] = subarray

What would be the right way of using slices to fill L the way I need?

Thanks!

pandas numpy arrays python
1 Answers

Your example is not very clear, but maybe you can adapt something from this code snippet. It looks to me like you are generating your L array, shape (x, y, z, a, b, c), by computing a slices, of shape (x, y, z, b, c), equivalent to (x, y, z, 1, b, c). Let me know if I am completely wrong.

import numpy as np

L = np.zeros((10, 10, 10, 2, 10, 10)) # shape (x, y, z, a, b, c)

def compute():
    return np.random.rand(10, 10, 10, 10, 10)  # shape (x, y, z, b, c)

for k in range(L.shape[3]):
    L[:, :, :, k, :, :] = compute() # Select slice of shape (x, y, z, b, c)

Basically, It computes a slice of the array (part of the array) and place it at the desired location.

One thing to note, an array of shape (x, y, z, a, b, c) can quickly get out of hand. For instance I naively tried to do L = np.zeros((100, 100, 100, 5, 100, 100)), resulting in a 373 Gb RAM allocation.. Depending on the size of your data, maybe you could work only on each slices and store them to disk when the others are not in use?

Following my comment, some snippet for you to get this dimension problem:

import numpy as np

L = np.zeros((10, 10, 10))
L.shape # (10, 10, 10)
L[:, 0, :].shape # (10, 10)
L[:, 0:3, :].shape # (10, 3, 10)

The slice on : selects all, the slice on x:y selets all from x to y and the slice on a specific index k selects only that 'line'/'column' (analogy for 2D), and thus returning an array of dimension n-1. In 2D, a line or column would be 1D.

user_83244800
Related