There is a function that can run a lot of time when the input is strange. For example, when the given SQL is very long, it costs a lot of time when running cursor.execute() in python sqlite3. I have two solutions to this problem.
- Using
Poolto create a process for each call, so I can end the process when it excelstimeoutseconds.
from multiprocessing import Pool
import time
def f(n):
print(n)
# a function that could run a lot of time, e.g., running a long or buggy SQL
time.sleep(10)
print("end f")
def timeout_f(timeout, f, *args, **kwargs):
pool = Pool(1)
try:
result = pool.apply_async(f, args, kwargs).get(timeout=timeout)
return_tuple = ('result', result)
except Exception as e:
return_tuple = ('exception', e)
finally:
pool.close()
pool.terminate()
pool.join()
return return_tuple
timeout_f(timeout=2, f=f, n=1)
print("end all")
However, the problem with this method is that creating a process every time consumes a lot of time compared to the serving function. For example, running a SQL with sqlite3 is fast at most times when the SQL is correct and straightforward.
- Another solution is using ThreadPool instead:
from multiprocessing.pool import ThreadPool
import time
def f(n):
print(n)
# a function that may run a lot of time, e.g., running a long or buggy SQL
time.sleep(10) # also could be time.sleep(0.001) when SQL is simple
print("end f")
def timeout_f(timeout, f, *args, **kwargs):
pool = ThreadPool(1)
try:
result = pool.apply_async(f, args, kwargs).get(timeout=timeout)
return_tuple = ('result', result)
except Exception as e:
return_tuple = ('exception', e)
finally:
pool.close()
pool.terminate()
pool.join()
return return_tuple
timeout_f(timeout=2, f=f, n=1)
print("end all")
However, I have to wait until all SQLs end (using join()), and in practice, I found many threads are created and never end because the SQL is long or buggy. This causes the main process stuck. (In above toy example, end all appears after end f )
So my question is how to speedup the first solution, or how to terminate the timeout threads in the second solution.