I am writing a Python genetic algorithm optimization library, and am finding my kernel is dying at random points for one of the problems I'm working on. The crashes have me pretty stumped, and any advice would be welcome.
Background:
I'm using Python 3.8.8. Among lots of other code, my library has the following objects:
- Individuals, these contain attributes like numpy arrays of the "genes", and other meta-data (name, fitness, etc.)
- Generations, contain a list of individuals, data about those individuals (e.g. their rank), and methods to pull data from the individuals.
- Analysis, which runs the actual optimization, as well as stores each generation and meta-data about the generations.
Some sample code snippets of questionable use: Generations
class Generation:
def __init__(self, population, genNumber):
# a list of all the individuals in a generaiton
self.population = population
self.size = len(population)
self.gen = genNumber
self.scores = [None]*self.size
self.ranks = [None]*self.size
self.best = None
def getCurrentGenome(self):
Genome = [None]*self.size
for ii, individual in enumerate(self.population):
Genome[ii] = individual.genotype
return Genome
...
Analysis object snippet
for ii in range(self.Ngen):
print('Generation ', ii + 1)
...
if self.recordGens == True:
gens[ii] = currentGen
...
The Problem:
The genetic algorithm solver sometimes crashes randomly for larger problems. By randomly, I mean the crash happens at different point in the analysis, (sometimes generation 100, sometimes generation 143, etc.) and if I'm lucky it won't crash. Note I'm using numpy for random numbers, and have fixed my random seed. The actual numbers being spit out in each generation appear to be the same between runs. When it crashes, it looks like this.
Running out of anaconda, there is a similar lack of errors:

What I've tried:
Debugging is hard because of the random nature of the crashes. However, I've determined:
- Crashes only occur if I store the generation objects.
- If instead of the generation object I store a random object like a list , e.g. ['a','b'], no crashes occur.
- The crashes don't appear to be related to the amount of data stored in each generation, a generation of 20 won't crash sooner than a generation of 100.
- If I write print statements at certain lines, the crash seems to occur at random, not at the same line each time.
- The amount of data I'm working with isn't that big - there are ~100 individuals in 1000 generations, and each individual only stores a few floats.
Question
I've been wondering about a memory problem, but I don't know much about memory management in python. Keeping an eye on the ram usage in task manager, it seems like I have memory to spare (50% ram usage). I'm wondering:
- Does anyone have any general suggestions for what might cause the crash? Any strategies for debugging when dealing with errors like this?
- If it is a memory problem, are there even any strategies I could use in Python to mitigate this?
