I have the following piece of code to estimate whether a new numpy array will fit in memory:
import numpy as np
from subprocess import Popen, PIPE
def getTotalMemory():
output, _= Popen(r'wmic ComputerSystem get TotalPhysicalMemory', stdout=PIPE).communicate()
val = ''.join([c for c in str(output) if c.isdigit()])
return int(val)
def getFreeMemory():
output, _= Popen(r'wmic OS get FreePhysicalMemory', stdout=PIPE).communicate()
val = ''.join([c for c in str(output) if c.isdigit()])
# FreePhysicalMemory returns kilobytes, hence the multiplication with 1024
return 1024 * int(val)
def estimateNumpySize(dim, square = False, nptype = np.float64):
# Make sure that the actual nptype is an np.dtype
nptype = np.dtype(nptype) if not isinstance(nptype, np.dtype) else nptype
# Determine the number of elements
numElem = dim if not square else dim * dim
# Return the value
return numElem * nptype.itemsize
# Get the memory sizes
freeMemory = getFreeMemory()
print('Total memory: {0}'.format(getTotalMemory()))
print('Free memory: {0}'.format(freeMemory))
for s in [3, 4, 5, 6, 7]:
estSize = estimateNumpySize(10**s, True)
print('\nEstimated array size: {0}'.format(estSize))
print('Will it fit?: {0}'.format(freeMemory > estSize))
try:
dummy = np.zeros((10**s, 10**s))
print('It fits! \nEstimated size equals actual size: {0}'.format(estSize == dummy.nbytes))
except MemoryError:
print('Array to large to fit in memory')
The output I get is the following, using both python 2.7 and python 3.7:
Total memory: 34276392960
Free memory: 23102312448
Estimated array size: 8000000
Will it fit?: True
It fits!
Estimated size equals actual size: True
Estimated array size: 800000000
Will it fit?: True
It fits!
Estimated size equals actual size: True
Estimated array size: 80000000000
Will it fit?: False
It fits!
Estimated size equals actual size: True
Estimated array size: 8000000000000
Will it fit?: False
Array to large to fit in memory
Estimated array size: 800000000000000
Will it fit?: False
Array to large to fit in memory
Which is more or less in line with my expectations, except for the fact that the third case apparently should not fit in memory, but the creation is no problem after all. I was wondering why that is the case?
The numbers returned by both memory functions are in line with the numbers shown in the Resource Monitor.
I know I do not strictly need to check before hand, since I can catch creation errors using a Try-Except block, but checking beforehand is cleaner in my opinion