I have a function crawl that take a natural number as input. Then I use concurrent.futures to speed of the loop of this function on rows, a list of integers. Because of privacy, I can not put the detail of function crawl here.
When I run my code, it seems to me that crawl stucks at some integer for an infinite amount of time. I would like to ask of we can add some lines of code to notify in case crawl starting to spend more than 5 minutes on an natural number.
Of course, the problem is that we want the notification after 5 minutes of running on this natural number, not after it finishes on this number (because it may stuck at this natural number forever).
rows = id[id['DONE'] == 0].index.tolist()
def crawl(i):
r = session.get(id.at[i, 'LINK'], headers = headers)
if r.status_code == 200:
id.at[i, 'DONE'] = 1
position = rows.index(i)
temp_text[position], temp_sound[position] = sub_crawl(r)
with concurrent.futures.ThreadPoolExecutor(max_workers = 15) as exe:
results = [exe.submit(crawl, i) for i in rows]