memory error while getting the combinations of a list

user_7138272 0
Viewed 132

I am trying to create combination of 34 items. when I limit max combinations to 10, there are no problems. however when I try to get the combination of more, I am having memory error. (MemoryError: Unable to allocate X GiB for an array with shape (X, Y) and data type float64) I am already using 64bit python, what would help with the iteration of these?

import pandas as pd
import itertools as it

header = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134]
combs= []
max_combinations = 20

for i in range(1, len(header)+1):
        if i < max_combinations:
            print (i)
            els = [list(x) for x in it.combinations(header, i)]
            combs.extend(els)
            
df1 = pd.DataFrame({0: combs})
pandas python-3.x python
1 Answers

You are really just running out of RAM. Not much you can do there.

A list by itself takes 56 B. Each 64-bit int in the list takes 8 B more.

I don't know the size of your array but say... If your list of lists is say 100,000 x 100,000 This will take about:

One list is 100000 x 8 + 56 = 805,600 B

1000 lists is then: 100000 x 805.6 kB = 80,560 MB

In a list = 80,560.000056 MB ~ 80G

If you have the ability to specify the datatype so you are using the minimum number of bits possible this could reduce your RAM usage slightly.

See the comment by chepner below for the memory required by a Python int.

user_145170580
Related