Python 3 multiprocessing.Process inside class?

Viewed 1109

I have some complex class A that computes data (large matrix calculations) while consuming input data from class B.

A itself uses multiple cores. However, when A needs the next chunk of data, it waits for quite some time since B runs in the same main thread.

Since A mainly uses the GPU for computations, I would like to have B collecting data concurrently on the CPU.

My latest approach was:

# every time *A* needs data
def some_computation_method(self):
    data = B.get_data()
    # start computations with data

...and B looks approximately like this:

class B(object):

    def __init__(self, ...):
        ...
        self._queue = multiprocessing.Queue(10)
        loader = multiprocessing.Process(target=self._concurrent_loader)

    def _concurrent_loader(self):
        while True:
            if not self._queue.full():
                # here: data loading from disk and pre-processing
                # that requires access to instance variables
                # like self.path, self.batch_size, ...
                self._queue.put(data_chunk)
            else:
                # don't eat CPU time if A is too busy to consume
                # the queue at the moment
                time.sleep(1)

    def get_data(self):
        return self._queue.get()

Could this approach be considered a "pythonic" solution?

Since I have not much experience with Python's multiprocessing module, I've built an easy/simplistic approach. However, it looks kind of "hacky" to me.

What would be a better solution to have a class B loading data from disk concurrently and supplying it via some queue, while the main thread runs heavy computations and consumes data from the queue from time to time?

1 Answers
Related