Kill method that is stuck via multithreading/multiprocessing

user_9136719 0
Viewed 54

I have a script that visits the website of a local weather station every hour and scrapes the current rainfall. This gets fed in my sprinkler server/database/etc. Sometimes there is a hiccup in the network connection or something on the side of the website. such an unfortunate makes the method hang indefinitely. This is not good for the stability of my program.

I've experimented with multiprocessing but I couldn't get it to work properly. Ideally it would launch the scraping module, and returns every second for a maximum of 10 so see if the method has stopped and produced an output. If the 10 sec is exceeded it should kill it and try the next hour. How would you tackle this issue?

My current script is this:

from multiprocessing import Process,Queue,Pipe
import time
import requests
import urllib.request
from bs4 import BeautifulSoup

url = "https://www.weerstationzoersel.be/weather2/index.php?p=10"
def get_rain():
    try:
        response = requests.get(url)
        responsestr = str(response)
        if "200" in responsestr:
            soup = BeautifulSoup(response.text, "html.parser")
            tags = soup.findAll('span')
            line_rain = str(tags[15])
            line_rain = line_rain[62::]
            rainfall = line_rain.rstrip("</span>")
            rainfall = round(float(rainfall.replace(',','.')),1)
    except:
        rainfall="error"
    return(rainfall)

if __name__ == '__main__':
    print(get_rain())
beautifulsoup multiprocessing python
2 Answers

The multiprocessing.pool.Pool class has the apply_async method that returns an AsyncResult object on which you can issue a get call to block for the return value. This method allows you to specify an optional timeout value such that if the task does not complete and has a result for you within the specified number of seconds a TimeoutError exception is thrown. However, the process in the pool is still continuing to run the submitted task. This no problem; you simply terminate the pool and all of its running processes.

Note that I modified get_rain to return the actual Exception object rather than the string "error" if an exception occurs:

from multiprocessing import Pool, TimeoutError
import requests
from bs4 import BeautifulSoup

def get_rain_worker():
    url = "https://www.weerstationzoersel.be/weather2/index.php?p=10"
    try:
        response = requests.get(url)
        # throw exception if error:
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        tags = soup.findAll('span')
        line_rain = str(tags[15])
        line_rain = line_rain[62::]
        rainfall = line_rain.rstrip("</span>")
        rainfall = round(float(rainfall.replace(',','.')),1)
    except Exception as e:
        # return the exception:
        rainfall = e
    return rainfall

def get_rain():
    # We just want a pool size of 1,
    # which will be automatically terminated at the end of the with block
    with Pool(1) as pool:
        async_result = pool.apply_async(get_rain_worker)
        try:
            # timeout after 10 seconds:
            rainfall = async_result.get(10)
        except TimeoutError as e:
            rainfall = e
    return rainfall

if __name__ == '__main__':
    rainfall = get_rain()
    if isinstance(rainfall, Exception):
        print('Got exception:', rainfall)
    else:
        print('rainfall = ', rainfall)

Prints:

rainfall =  0.0

A Simpler Approach?

I would think, however, that the only hang-up would be on doing the GET request, which does take an optional timeout argument. In fact, it is recommended that this argument be specified for all production code. So the following code may be sufficient for your needs:

import requests
from bs4 import BeautifulSoup

def get_rain():
    url = "https://www.weerstationzoersel.be/weather2/index.php?p=10"
    try:
        response = requests.get(url, timeout=10)
        # throw exception if error:
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        tags = soup.findAll('span')
        line_rain = str(tags[15])
        line_rain = line_rain[62::]
        rainfall = line_rain.rstrip("</span>")
        rainfall = round(float(rainfall.replace(',','.')),1)
    except Exception as e:
        # return the exception:
        rainfall = e
    return rainfall

if __name__ == '__main__':
    rainfall = get_rain()
    if isinstance(rainfall, Exception):
        print('Got exception:', rainfall)
    else:
        print('rainfall = ', rainfall)
user_28237190

It might work to set a signal alarm to go off if the connection hangs for too long. Something like this:

import signal

def timeout():
    raise TimeoutError

signal.signal(signal.SIGALRM, timeout)
signal.setitimer(signal.ITIMER_REAL, 10)

followed by your existing try block. If the get() doesn't complete in 10 seconds, your try block will get a TimeoutError exception.

If you do complete the get() you can call setitimer() with a seconds count of 0 to remove the previous timer.

user_138432680
Related