Is it possible to do multiple different concatenates with one call?

user_13200853 0
Viewed 81

I have 3 independent numpy.array and need to concatenate them with another 3 within a loop thousands of times. For the sake of performance, I would like to know if there is some way to do these 3 concatenate with only one numpy.concatenate() call.

Right now I have:

arr1 = np.concatenate([arr1, _arr1])
arr2 = np.concatenate([arr2, _arr2])
arr3 = np.concatenate([arr3, _arr3])

And I would like to get the same result with just one call, for example:

arr = [arr1, arr2, arr3]
_arr = [_arr1, _arr2, _arr3]
arr = np.concatenate([arr, _arr])

Is that somehow possible? If not, what would be the best approach?

Example of an input and desired output:

#Inputs
arr1 = [1,2]
arr2 = [3,4]
arr3 = [5,6]
_arr1 = [-1,-2]
_arr2 = [-3,-4]
_arr3 = [-5,-6]
#Output
arr = [[1,2,-1,-2], [3,4,-3,-4], [5,6,-5,-6]]
concatenation numpy python
2 Answers

Optimization is best when you don't re-allocate memory.

So you should set up one or more arrays for everything you need to do within the loop. If you can get everything you need into one big array you may not even need to do a concatenate operation.

Here's an example of the technique

# Declare one array for all data
data = np.zeros((3,6))

# Define sub-arrays
arr1 = data[0,:3]
_arr1 = data[0,3:]
arr2 = data[1,:3]
_arr2 = data[1,3:]
arr3 = data[2,:3]
_arr3 = data[2,3:]

for i in range(5):

    # New values from somewhere
    arr1[:] = np.random.randn(3)                                                      
    arr2[:] = np.random.randn(3)                                                      
    arr3[:] = np.random.randn(3)                                                     
    _arr1[:] = np.random.randn(3)
    _arr2[:] = np.random.randn(3)
    _arr3[:] = np.random.randn(3)

    # Combined arrays
    print(data[0])  # same as np.concatenate([arr1, _arr1])
    print(data[1])  # same as np.concatenate([arr2, _arr2])
    print(data[2])  # same as np.concatenate([arr3, _arr3])

Note that when you do assignment with an index like [:] it does not create a new array. Instead it assigns the values to the existing array. This should be faster (I didn't check) since it doesn't need to allocate new memory each loop.

user_16095140

In the end I have followed this advice:

Optimization is best when you don't re-allocate memory.

So you should set up one or more arrays for everything you need to do within the loop. If you can get everything you need into one big array you may not even need to do a concatenate operation.

I pre-allocate the arrays heuristically to the worst case and instead of concatenating them I copy the values from the temporal arrays. With this approach I achieved to double the performance.

user_132008530
Related