flatten large list of lists, deallocate along the way

user_353337 0
Viewed 201

I have a list of list of m-by-n NumPy arrays that I get from a third-party library, e.g.,

import numpy

x0 = numpy.random.rand(1000, 1000)  # x_i likewise
a =  [[x0], [x1, x2], [x3, x4, x5]]   # and more

I now need to flatten this structure, e.g.,

b = numpy.concatenate(a)
# b =  [x0, x1, x2, x3, x4, x5]

The arrays x are however so large that both a and b don't fit into memory at the same time, so Python is oom-killed at concatenate.

Can I flatten a into a contiguous numpy array while deallocating the elements of a along the way?

memory-management numpy arrays python
1 Answers

So a is a list of lists of arrays, where all the arrays have the same shape?

concatenate takes a list of arrays, and joins them on the desired axis. It's compiled so it's hard to say what it's doing with the lists of lists etc. This test shows it behaves as though the list of lists was flat list. But I can't say whether it flattens the lists or does nested concatenates.

In [339]: x0=np.ones((3,4),int)
In [340]: a=[[x0],[x0,x0],[x0,x0,x0]]
In [341]: np.concatenate(a)
In [342]: _.shape
Out[342]: (6, 3, 4)

The nested list can be flattened with itertools

In [343]: import itertools
In [344]: b=list(itertools.chain(*a))

Being a list operation it shouldn't affect memory use much, since it's just playing with pointers, not making new arrays.

It may be possible to concatenate chunks (sub lists), and delete elements from the lists. But for this to be effective we have to ensure that all references to individual arrays are gone, other wise they won't be garbage collected.

But I'm not going to experiment with arrays that might give me memory full errors.

user_9019250
Related