I have made a program consisting of several classes and files. It reads some input from USB, processes it and writes it to a CSV-file. It's an ongoing process spitting out CSV-files several times a day based on user configured time periods where the program monitors the USB-input.
I am writing another script to read each CSV-file, when all the USB-input for the period has been written to the file.
Since I don't want to interrupt the ongoing USB-hid-reading process, I would like to utilize the multiprocessing functionality in Python and spawn a temporary handler each time a file is ready for processing.
So far I've made this little test script that takes the location of the CSV-file as input and prints out the content, while the parent process keeps on running.
import multiprocessing
import csv
class Handler(multiprocessing.Process):
def __init__(self, period: dict = None, csv_file: str = None):
super().__init__()
self.period = period # dict containing configuration from parent
self.csv_file = csv_file
def run(self):
with open(self.csv_file, newline='') as csvfile:
reader = csv.reader(csvfile, delimiter=';')
for row in reader:
print(', '.join(row))
The child process is started from and instance of a class called Period. Each period represents a user defined time period in witch to read USB-data. When a CSV-file is closed a child process is started from the perioed-instance in the following way:
p = handler.Handler(self.period,self.csv_path)
p.start()
# the program continues to execute, writing more csv-files and starting more handlers
The above seems to work as intended. However in theory a lot of processes could be spawned this way.
Ideally I would like to feed CSV-paths together with the period-configuration from each period-instance into a queue which would be processed by just a single child process across the entire app and not a new child process for each period.
I have read about queues, pipes and pools but I'm not really sure how to go about it. So I'm looking for any input to help me on the right track.
The processing of the CSV-files are not time critical, the reading from the USB-device is.
Also the waste majority of multiprocessing examples I've seen are using a call from __main__ Something to do with how Windows cannot fork child processes the same way as Unix.
I have one __main__ method in my program (in main.py) and it is really not ideal to implement any of the above logic in there. That is why I ended up overriding instead. Not sure if that is the way to do it though.