How to solve memory leak with python multiprocessing and scipy periodogram

Question

I'm attempting to debug what appears to be a memory leak in the interplay between scipy.signal.periodogram and python's multiprocessing. Here is a minimal example:

#!/usr/bin/env python3
import numpy as np
from scipy import signal
from memory_profiler import profile
import gc

fs = 30 # Sampling rate

def genSpec(args):
    time = np.arange(60*30) / fs # 1 minute at 30 hz
    freq = 65
    x = np.sin(2*np.pi*freq*time)
    w = fs*10 # Window length
    spec = [signal.periodogram(x[i:i+w], fs=fs, nfft=int(fs/.001)) for i in range(len(x)-w+1)]
    return spec

def genNoSpec(args):
    spec = genSpec(args)
    return None

def genSpecNP(args):
    spec = genSpec(args)
    return np.array(spec)

def doMultiprocess(numItems, func):
    from multiprocessing import Pool
    with Pool() as p:
        list(p.imap_unordered(func, range(numItems))) # Do nothing with it

@profile
def gimmeData(numItems):
    print('Generating without spec')
    doMultiprocess(numItems, genNoSpec)
    print('Generating spec wrapped in np.array')
    doMultiprocess(numItems, genSpecNP)
    print('Generating with spec')
    doMultiprocess(numItems, genSpec)
    print('Generation completed')
    gc.collect()
    print('Garbage collected')

gimmeData(16)
input(f'Press [Enter] to exit')

When run, the memory profiler produces:

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    30     76.5 MiB     76.5 MiB           1   @profile
    31                                         def gimmeData(numItems):
    32     76.5 MiB      0.0 MiB           1       print('Generating without spec')
    33     77.4 MiB      0.9 MiB           1       doMultiprocess(numItems, genNoSpec)
    34     77.4 MiB      0.0 MiB           1       print('Generating spec wrapped in np.array')
    35     78.4 MiB      1.0 MiB           1       doMultiprocess(numItems, genSpecNP)
    36     78.4 MiB      0.0 MiB           1       print('Generating with spec')
    37   5389.3 MiB   5310.9 MiB           1       doMultiprocess(numItems, genSpec)
    38   5389.3 MiB      0.0 MiB           1       print('Generation completed')
    39   5384.7 MiB     -4.7 MiB           1       gc.collect()
    40   5384.7 MiB      0.0 MiB           1       print('Garbage collected')

It seems that:

The spectrogram generated by scipy.signal.periodogram doesn't get garbage collected if it is passed from the child thread back to the parent (as in genSpec), even if it then immediately goes out of scope.
If it isn't passed back to the parent, then it's collected (as in genNoSpec)
If it's first wrapped in a numpy array, then it's collected as well (as in genSpecNP).

This last one makes me think that numpy's data conversions strip away whatever the culprit is that was placed there by the periodogram call, but beyond that I'm at a loss. Anyone know why this might be?

Other info: I'm running an anaconda environment with python 3.7.6, scipy version 1.6.2, and have tested this both on Ubuntu 18.04.6 and Gentoo. EDIT: I've had the same behavior on Arch Linux with python 3.10.5 and scipy 1.8.1.

EDIT: Made example more minimal by replacing signal.spectrogram with wrapping periodogram in numpy array

How to solve memory leak with python multiprocessing and scipy periodogram

0 Answers