For some context: I am currently trying to write a tool that automatically grades (over 100) submissions. Each submission comes in the form of a solve.py file that contains a solve_exercise function. Also, each submission is stored in a separate directory on my file system. The grading of a submission is done by calling solve_exercise on several exercises and checking the solution. After grading a submission, I want to go on to grade the next one. As such, I want to import a module named solve over a 100 times (once for each submission); however, each time, the module's implementation is to be fetched from a different location (and the previously used implementations are irrelevant and should be discarded or at least ignored). To speed up the process, I wanted to use multiprocessing.
Now, I have the following code, which, based on some testing, seems to work. However, I am not sure if this is the proper way of doing it and I am not accidentally introducing unwanted side effects or reusing the same solve implementation multiple times. (I tried to make the code below as minimal as possible and only include the things relevant to my question.)
def grade_submission(submission_dirpath : str, submission_identifier : str):
submission_solve_module_filepath = os.path.join(submission_dirpath, "solve.py")
# Import solve module
submission_solve_module_spec = importlib.util.spec_from_file_location("solve", submission_solve_module_filepath)
submission_solve_module = importlib.util.module_from_spec(submission_solve_module_spec)
sys.modules["solve"] = submission_solve_module
submission_solve_module_spec.loader.exec_module(submission_solve_module)
# Grade submission using imported solve module...
if __name__ == "__main__":
# Parse command line arguments
argparser = argparse.ArgumentParser(description="")
argparser.add_argument("sdp", type=str, help="Path to directory that contains all submission directories.")
args = argparser.parse_args()
# Get subdirectories in submissions directory
dir_path, subdir_names, _ = next(os.walk(args.sdp))
# Construct list of argument tuples corresponding to submissions (i.e., the grade_submission function should be called once for each of these tuples)
submission_arguments = []
for subdir_name in subdir_names:
# Create path to submission subdirectory
subdir_path = os.path.join(dir_path, subdir_name)
submission_arguments.append((subdir_path, subdir_name))
# Parallelize grading
with multiprocessing.Pool() as pool:
pool.starmap(grade_submission, submission_arguments)
pool.close()
pool.join()
Is this a proper way of replacing the imported solve module each time a process proceeds to the next submission?