os.system() leads to invalid console output of print() python function even under mutex

user_15452785 0
Viewed 115

I made a script which calls some program via os.system() inside a ThreadPool and prints results of work to console. Simplified code looks this way:

import os
from concurrent.futures import ThreadPoolExecutor, as_completed
import multiprocessing
from threading import Lock

print_mutex = Lock()
def print_safe(text):
    with print_mutex:
        print(text, flush=True)

def print_smth():
    res = os.system("")
    print_safe("1")

with ThreadPoolExecutor(max_workers = multiprocessing.cpu_count()) as thread_pool:
    future_results = [thread_pool.submit(print_smth) for i in range(40)]

    for future in future_results:
        future.result()

But in console output I receive unexpected result like that:

Console output

If I comment res = os.system("") call, then output becomes valid.

Why does os.system() influences such way on print function?

What is the correct way to implement process calls to avoid such issue?

os.system threadpool python-3.x windows python
1 Answers

What is happening:

Threadpool creates a pool of threads that do work in parallel. However, your main thread is still a single thread. If you have a job that takes no time (less than the cycle of the loop calling a function itself) it will finish in the time it takes to call the next function, it will appear synchronous.

os.system is taking some sort of time on your pc (this can change depending on speed, OS, and number of CPU) so threads are ending simultaneously, and due to the delay for the work of calling os.system they are printing next to each other since they printed in the same single threaded cycle of the main app (the one printing to console).

If your function does actual work, you would see the results you were expecting.

user_35910140
Related