Correct way to handle failed request - python

user_16896291 0
Viewed 49

I am using requests library in python.

def get_response(url):
    try:
        session = requests.Session()
        retry = Retry(connect=3, backoff_factor=0.5)  
        adapter = HTTPAdapter(max_retries=retry)
        session.mount('https://', adapter)
        session.mount('http://', adapter)                  
        response = session.get(url)
        return response
    except Exception as e:
        log.error(f"{str(e)}")

def get_soup(html):
    try:
        soup = BeautifulSoup(html, 'html.parser')
        return soup 
    except Exception as e:
        log.error(f"{str(e)}")

I am using get_response function to make requests. When server is up, try block return me the response and I use it in further code... but when server is down, except block returns None and my whole code fails after that.

How can I handle this situation, the whole code relies on the response.. like response.text, response.status_code etc.

for example:

response = get_response(some_url)
html = response.text
soup = get_soup(html)

so when actual response is received from try block... the code works fine after that. but in case, lets assume server is down, so except block will run and return None (here I am not returning explicitly). So this html = response.text will fail saying NoneType has no attribute text

python-requests python
1 Answers

I Will implement the program this way.

def get_response(url):
    try:
        session = requests.Session()
        retry = Retry(connect=3, backoff_factor=0.5)
        adapter = HTTPAdapter(max_retries=retry)
        session.mount('https://', adapter)
        session.mount('http://', adapter)
        response = session.get(url)
        return response
    except Exception as e:
        log.error(f"{str(e)}")


def get_soup(html):
    try:
        soup = BeautifulSoup(html, 'html.parser')
        return soup
    except Exception as e:
        log.error(f"{str(e)}")


response = get_response(some_url)
if response:
    html = response.text
    soup = get_soup(html)
else:
    print('Error: Some error message')

At if block if the response value is None than the else statement will run.

user_162097960
Related