My main process couldn't continue after calling join for the child processes, these are the long run child processes, and I can confirm that they finished running, because I check the queue result before doing the join. So the main process didn't run anything for almost 10 minutes, until I decide to close the terminal window, at this moment it run a little more commands, I know this because I logged the command into a debug file. I'd like to know what kind of behavior is this, why it continues at the moment I close the terminal.
P.S. Sometimes this bug happened, sometimes the code works fine until the end. I added the code related to the parallel part below. If the bug happened, normally it's right after I call join on a child process, but I knew that child process is supposed to be finished already thanks to status_dict
with multiprocessing.Manager() as manager:
semaphore = manager.Semaphore(value=param_config.max_process)
status_dict = manager.dict()
l_id = -1
list_process = []
while l_id <= max_link_id:
proc_amount = 0
if l_id >= max_link_id:
l_id += 1
while proc_amount < param_config.batch_size and l_id < max_link_id:
l_id += 1
proc_amount += 1
semaphore.acquire()
process = TabuProcess(result_queue, l_id, map_slot, map_dual_values, map_C_pi, map_lightpath_id.values(), \
graph, map_config_generator[l_id].map_route_link, map_config_generator[l_id].map_request_route, \
semaphore, status_dict)
list_process.append(process)
process.start()
while not result_queue.empty():
lid, map_slot, _reduce_cost = result_queue.get()
if l_id < max_link_id:
t_list = list(list_process)
for proc in t_list:
if status_dict[proc.link_id]:
proc.join()
list_process.remove(proc)
elif l_id == max_link_id:
while len(list_process) > 0:
t_list = list(list_process)
for proc in t_list:
if status_dict[proc.link_id]:
list_process.remove(proc)
proc.join()
else:
if len(list_process) > 0:
time.sleep(60)