Implement prefetch with python multiprocessing

Viewed 163

I want to implement prefetch functionality for a customized class using multiprocessing of python.

Suppose we have the following class Sequence which can get batch of data by using index with __getitem__ method

class Sequence:
    
    def __init__(self):
        """ class initialization """

    def __getitem__(self, idx):
        """ get data by index, this step may cost some time """

and a very time-comsuming function run_experiment to process the data returned by Sequence.

Typically I will use for-loop as follows

for i in range(1000):
    data = Sequence[i]
    output = run_experiment(data)

However, since run_experiment is time-consuming, I would expect that Sequence can pre-preparing some data for future uses while run_experiment is still running. Naive idea will be using queue to collect data returned by Sequence in advance and directly get data from queue to feed run_experiment. However, I have no idea how to implement such functionality in Sequence while not changing too much of its API. All I need is a simple demo of solution for my problem.

0 Answers
Related