How to notify when the loop starts spending more than 5 minutes on an index?

Viewed 39

I have a function crawl that take a natural number as input. Then I use concurrent.futures to speed of the loop of this function on rows, a list of integers. Because of privacy, I can not put the detail of function crawl here.

When I run my code, it seems to me that crawl stucks at some integer for an infinite amount of time. I would like to ask of we can add some lines of code to notify in case crawl starting to spend more than 5 minutes on an natural number.

Of course, the problem is that we want the notification after 5 minutes of running on this natural number, not after it finishes on this number (because it may stuck at this natural number forever).

rows = id[id['DONE'] == 0].index.tolist()

def crawl(i):
    r = session.get(id.at[i, 'LINK'], headers = headers)
    if r.status_code == 200:
        id.at[i, 'DONE'] = 1
        position = rows.index(i)
        temp_text[position], temp_sound[position] = sub_crawl(r)

with concurrent.futures.ThreadPoolExecutor(max_workers = 15) as exe:
    results = [exe.submit(crawl, i) for i in rows]
0 Answers
Related