Using multiprocessing to return multiple instances of a class inside a shared container class

Viewed 57

I have a very text/data heavy class method that I would like to break up across 36 cores using the Python multiprocessing package. Some of this is ported code from an R project I had do roughly the same. Using R it was relatively simple, I split the data frame into N (36 in this example) chunks and then ran a multiprocessing foreach loop over the function with the foreach loop index iterating across the broken up data frame.

In the code below, an instance of this class calls the bottom method buildAllPatients which extracts all the patent IDs (patidList) and passes them to the buildPatientsFromPatid. In turn, that method calls the private method __populatePatient for each entry of patidList. It creates an object of class Patient, executes a collection of getters and setters, and then returns the Patient object to add it to a container class instance called Patients.

I want to add a pool of workers inside the buildPatientsFromPatid method, essentially turning the for loop into some multiprocessing code. Each worker needs to return a unique Patient object having called __populatePatient and then return it to the shared patients object.

Essentially, this is helpful when there are somewhere in the order of 500,000 patient IDs, each turning into a Patient object and having a large number of getters and setters inside __populatePatient.

def __populatePatient(self, featureList, patidStr, indDF):
    #instantiates some objects, executes some getters and setters
    #returns the object patient to be added to the collector class Patients
    return patient

def buildPatientsFromPatid(self, patidList, featureList) -> Patients:
    # go through each entry of patidList, build a new Patient and assigned to a PatientsClass instance
    patients = Patients()

    #Iterate over each patient ID calling the private function
    #to build a Patient object, return it and add to the larger
    #Patients object
    for patidStr in patidList:
        patients.addPatient(self.__buildPatientFromPatid(patidStr, featureList))
    return patients


# This is called FIRST....
def buildAllPatients(self, featureList):
    patidList = self.ioDF.patid.unique()
    patients = self.buildPatientsFromPatid(patidList, featureList)
    return patients

Update #1 I wanted to add that this code is far away from the main function of the software. All examples I've found dealing with multiprocessing and object orientation have some element of multiprocessing executed from inside the main function.

0 Answers
Related