How to start a python process efficiently or how to end a thread?

Viewed 20

There is a function that can run a lot of time when the input is strange. For example, when the given SQL is very long, it costs a lot of time when running cursor.execute() in python sqlite3. I have two solutions to this problem.

  1. Using Pool to create a process for each call, so I can end the process when it excels timeout seconds.
from multiprocessing import Pool
import time


def f(n):
    print(n)
    # a function that could run a lot of time, e.g., running a long or buggy SQL
    time.sleep(10)
    print("end f")


def timeout_f(timeout, f, *args, **kwargs):
    pool = Pool(1)
    try:
        result = pool.apply_async(f, args, kwargs).get(timeout=timeout)
        return_tuple = ('result', result)
    except Exception as e:
        return_tuple = ('exception', e)
    finally:
        pool.close()
        pool.terminate()
        pool.join()
        return return_tuple


timeout_f(timeout=2, f=f, n=1)
print("end all")

However, the problem with this method is that creating a process every time consumes a lot of time compared to the serving function. For example, running a SQL with sqlite3 is fast at most times when the SQL is correct and straightforward.

  1. Another solution is using ThreadPool instead:
from multiprocessing.pool import ThreadPool
import time


def f(n):
    print(n)
    # a function that may run a lot of time, e.g., running a long or buggy SQL
    time.sleep(10)  # also could be time.sleep(0.001) when SQL is simple
    print("end f")


def timeout_f(timeout, f, *args, **kwargs):
    pool = ThreadPool(1)
    try:
        result = pool.apply_async(f, args, kwargs).get(timeout=timeout)
        return_tuple = ('result', result)
    except Exception as e:
        return_tuple = ('exception', e)
    finally:
        pool.close()
        pool.terminate()
        pool.join()
        return return_tuple


timeout_f(timeout=2, f=f, n=1)
print("end all")

However, I have to wait until all SQLs end (using join()), and in practice, I found many threads are created and never end because the SQL is long or buggy. This causes the main process stuck. (In above toy example, end all appears after end f )

So my question is how to speedup the first solution, or how to terminate the timeout threads in the second solution.

0 Answers
Related