I want to implement prefetch functionality for a customized class using multiprocessing of python.
Suppose we have the following class Sequence which can get batch of data by using index with __getitem__ method
class Sequence:
def __init__(self):
""" class initialization """
def __getitem__(self, idx):
""" get data by index, this step may cost some time """
and a very time-comsuming function run_experiment to process the data returned by Sequence.
Typically I will use for-loop as follows
for i in range(1000):
data = Sequence[i]
output = run_experiment(data)
However, since run_experiment is time-consuming, I would expect that Sequence can pre-preparing some data for future uses while run_experiment is still running. Naive idea will be using queue to collect data returned by Sequence in advance and directly get data from queue to feed run_experiment. However, I have no idea how to implement such functionality in Sequence while not changing too much of its API. All I need is a simple demo of solution for my problem.