HTTP Error 503 when trying to download data from the site

Question

Hi i am trying to get table data from site GOGOV. It provides information on vaccination that interests me.

I wrote the following list:

query_dataset = 'https://gogov.ru/articles/covid-v-stats'
name_source = 'GHR_Y00011440'
sources = 'https://gogov.ru/covid-v-stats/'

data_list_region = [['RU-MOW', 'msk'], ['RU-MOS', 'mo'], ['RU-SPE', 'spb'], ['RU-KDA', 'krd'], ['RU-ROS', 'rst'], ['RU-BA', 'bash'],
                    ['RU-SVE', 'svo'], ['RU-TT', 'rt'], ['RU-SAM', 'smr'], ['RU-NIZ', 'ngo'], ['RU-BEL', 'blg'], ['RU-NVS', 'nso'],
                    ['RU-CHE', 'chlb'], ['RU-KEM', 'kmr'], ['RU-VGG', 'vlg'], ['RU-KYA', 'kry'], ['RU-SAR', 'srt'],
                    ['RU-IRK', 'irk'], ['RU-VOR', 'vrzh'], ['RU-STA', 'srp'], ['RU-KHM', 'hmao'], ['RU-ALT', 'altkr'],
                    ['RU-PER', 'prm'], ['RU-ORE', 'orb'], ['RU-CR', 'crimea'], ['RU-TYU', 'tmn'], ['RU-TUL', 'tla'], ['RU-OMS', 'omsk'],
                    ['RU-CE', 'che'], ['RU-LEN', 'lo'], ['RU-PRI', 'prim'], ['RU-LIP', 'lpts'], ['RU-KIR', 'krv'], ['RU-BRY', 'brn'],
                    ['RU-SA', 'yak'], ['RU-KRS', 'krsk'], ['RU-TVE', 'tvr'], ['RU-PNZ', 'pnz'], ['RU-ZAB', 'zbk'], ['RU-IVA', 'ivnv'],
                    ['RU-UD', 'udm'], ['RU-BU', 'brt'], ['RU-AST', 'ast'],
                    ['RU-ULY', 'uln'], ['RU-MO', 'mrd'], ['RU-TAM', 'tmb'], ['RU-KHA', 'khbr'], ['RU-KGD', 'klg'],
                    ['RU-CU', 'chvsh'], ['RU-YAR', 'yar'], ['RU-DA', 'dg'], ['RU-TOM', 'tms'], ['RU-ARK', 'arkh'],
                    ['RU-VLG', 'vlgd'], ['RU-KLU', 'klga'], ['RU-RYA', 'rzn'], ['RU-KGN', 'krg'], ['RU-AMU', 'amr'], ['RU-KO', 'kmi'],
                    ['RU-VLA', 'vld'], ['RU-SMO', 'sml'], ['RU-MUR', 'mrm'],
                    ['RU-ORL', 'orl'], ['RU-SAK', 'skhl'], ['RU-YAN', 'yam'], ['RU-KR', 'krl'],
                    ['RU-KOS', 'ktm'], ['RU-NGR', 'nvg'], ['RU-KK', 'khks'], ['RU-SEV', 'sev'], ['RU-PSK', 'psk'], ['RU-ME', 'mrl'],
                    ['RU-KB', 'kbr'], ['RU-IN', 'ing'],
                    ['RU-SE', 'ost'], ['RU-TY', 'tva'],
                    ['RU-KC', 'krch'], ['RU-AD', 'adg'], ['RU-KAM', 'kmch'], ['RU-KL', 'klm'], ['RU-ALT', 'alt'], ['RU-MAG', 'mgd'],
                    ['RU-YEV', 'eao'], ['RU-CHU', 'chuk'], ['RU-NEN', 'nao']]

Function to download:

def save(source, name, directory, region_code = None):
      ru_region_save = pd.read_html(source, header=0)
      ru_region_save = ru_region_save[0]
      if region_code != None:
          ru_region_save['region_code'] = region_code
      ru_region_save.to_csv(directory + name, index=False)

save(source=query_dataset, name='now.csv', directory=directory)

I get the following error, which I think makes it clear that the site sees me as a bot. Even on the site, with an intensive transition from the page, a captcha may appear.

urllib.error.HTTPError: HTTP Error 503: Service Temporarily Unavailable

Process finished with exit code 1

I manage to get 3 files from 3 pages and then I get an error. How can I solve this problem?

user_415016 · Answer

HTTP 503 means the server is busy and cant serve you at the moment. You need to try and pause for few seconds between HTTP call to the server.

user_9192284 · Answer

Try:

import pandas as pd
from time import sleep  # <<---

query_dataset = 'https://gogov.ru/articles/covid-v-stats'
name_source = 'GHR_Y00011440'
sources = 'https://gogov.ru/covid-v-stats/'

data_list_region = [['RU-MOW', 'msk'], ['RU-MOS', 'mo'], ['RU-SPE', 'spb'], ['RU-KDA', 'krd'], ['RU-ROS', 'rst'], ['RU-BA', 'bash'],
                    ['RU-SVE', 'svo'], ['RU-TT', 'rt'], ['RU-SAM', 'smr'], ['RU-NIZ', 'ngo'], ['RU-BEL', 'blg'], ['RU-NVS', 'nso'],
                    ['RU-CHE', 'chlb'], ['RU-KEM', 'kmr'], ['RU-VGG', 'vlg'], ['RU-KYA', 'kry'], ['RU-SAR', 'srt'],
                    ['RU-IRK', 'irk'], ['RU-VOR', 'vrzh'], ['RU-STA', 'srp'], ['RU-KHM', 'hmao'], ['RU-ALT', 'altkr'],
                    ['RU-PER', 'prm'], ['RU-ORE', 'orb'], ['RU-CR', 'crimea'], ['RU-TYU', 'tmn'], ['RU-TUL', 'tla'], ['RU-OMS', 'omsk'],
                    ['RU-CE', 'che'], ['RU-LEN', 'lo'], ['RU-PRI', 'prim'], ['RU-LIP', 'lpts'], ['RU-KIR', 'krv'], ['RU-BRY', 'brn'],
                    ['RU-SA', 'yak'], ['RU-KRS', 'krsk'], ['RU-TVE', 'tvr'], ['RU-PNZ', 'pnz'], ['RU-ZAB', 'zbk'], ['RU-IVA', 'ivnv'],
                    ['RU-UD', 'udm'], ['RU-BU', 'brt'], ['RU-AST', 'ast'],
                    ['RU-ULY', 'uln'], ['RU-MO', 'mrd'], ['RU-TAM', 'tmb'], ['RU-KHA', 'khbr'], ['RU-KGD', 'klg'],
                    ['RU-CU', 'chvsh'], ['RU-YAR', 'yar'], ['RU-DA', 'dg'], ['RU-TOM', 'tms'], ['RU-ARK', 'arkh'],
                    ['RU-VLG', 'vlgd'], ['RU-KLU', 'klga'], ['RU-RYA', 'rzn'], ['RU-KGN', 'krg'], ['RU-AMU', 'amr'], ['RU-KO', 'kmi'],
                    ['RU-VLA', 'vld'], ['RU-SMO', 'sml'], ['RU-MUR', 'mrm'],
                    ['RU-ORL', 'orl'], ['RU-SAK', 'skhl'], ['RU-YAN', 'yam'], ['RU-KR', 'krl'],
                    ['RU-KOS', 'ktm'], ['RU-NGR', 'nvg'], ['RU-KK', 'khks'], ['RU-SEV', 'sev'], ['RU-PSK', 'psk'], ['RU-ME', 'mrl'],
                    ['RU-KB', 'kbr'], ['RU-IN', 'ing'],
                    ['RU-SE', 'ost'], ['RU-TY', 'tva'],
                    ['RU-KC', 'krch'], ['RU-AD', 'adg'], ['RU-KAM', 'kmch'], ['RU-KL', 'klm'], ['RU-ALT', 'alt'], ['RU-MAG', 'mgd'],
                    ['RU-YEV', 'eao'], ['RU-CHU', 'chuk'], ['RU-NEN', 'nao']]

def save(source, name, directory, region_code = None):
    ru_region_save = pd.read_html(source, header=0)

    # allow the site to pause for breath
    sleep(5) # <<---

    ru_region_save = ru_region_save[0]
    if region_code != None:
        ru_region_save['region_code'] = region_code
    ru_region_save.to_csv(directory + name, index=False)

# I had to use this, but use whatever you need...    
directory = './'

save(source=query_dataset, name='now.csv', directory=directory)

Output:

Additional:

I can only guess at what to try if the site still fails. Maybe using requests and setting headers with something like:

import pandas as pd
from time import sleep  # <<--
import requests         # <<--

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3"
}  # <<--

query_dataset = 'https://gogov.ru/articles/covid-v-stats'
name_source = 'GHR_Y00011440'
sources = 'https://gogov.ru/covid-v-stats/'

data_list_region = [['RU-MOW', 'msk'], ['RU-MOS', 'mo'], ['RU-SPE', 'spb'], ['RU-KDA', 'krd'], ['RU-ROS', 'rst'], ['RU-BA', 'bash'],
                    ['RU-SVE', 'svo'], ['RU-TT', 'rt'], ['RU-SAM', 'smr'], ['RU-NIZ', 'ngo'], ['RU-BEL', 'blg'], ['RU-NVS', 'nso'],
                    ['RU-CHE', 'chlb'], ['RU-KEM', 'kmr'], ['RU-VGG', 'vlg'], ['RU-KYA', 'kry'], ['RU-SAR', 'srt'],
                    ['RU-IRK', 'irk'], ['RU-VOR', 'vrzh'], ['RU-STA', 'srp'], ['RU-KHM', 'hmao'], ['RU-ALT', 'altkr'],
                    ['RU-PER', 'prm'], ['RU-ORE', 'orb'], ['RU-CR', 'crimea'], ['RU-TYU', 'tmn'], ['RU-TUL', 'tla'], ['RU-OMS', 'omsk'],
                    ['RU-CE', 'che'], ['RU-LEN', 'lo'], ['RU-PRI', 'prim'], ['RU-LIP', 'lpts'], ['RU-KIR', 'krv'], ['RU-BRY', 'brn'],
                    ['RU-SA', 'yak'], ['RU-KRS', 'krsk'], ['RU-TVE', 'tvr'], ['RU-PNZ', 'pnz'], ['RU-ZAB', 'zbk'], ['RU-IVA', 'ivnv'],
                    ['RU-UD', 'udm'], ['RU-BU', 'brt'], ['RU-AST', 'ast'],
                    ['RU-ULY', 'uln'], ['RU-MO', 'mrd'], ['RU-TAM', 'tmb'], ['RU-KHA', 'khbr'], ['RU-KGD', 'klg'],
                    ['RU-CU', 'chvsh'], ['RU-YAR', 'yar'], ['RU-DA', 'dg'], ['RU-TOM', 'tms'], ['RU-ARK', 'arkh'],
                    ['RU-VLG', 'vlgd'], ['RU-KLU', 'klga'], ['RU-RYA', 'rzn'], ['RU-KGN', 'krg'], ['RU-AMU', 'amr'], ['RU-KO', 'kmi'],
                    ['RU-VLA', 'vld'], ['RU-SMO', 'sml'], ['RU-MUR', 'mrm'],
                    ['RU-ORL', 'orl'], ['RU-SAK', 'skhl'], ['RU-YAN', 'yam'], ['RU-KR', 'krl'],
                    ['RU-KOS', 'ktm'], ['RU-NGR', 'nvg'], ['RU-KK', 'khks'], ['RU-SEV', 'sev'], ['RU-PSK', 'psk'], ['RU-ME', 'mrl'],
                    ['RU-KB', 'kbr'], ['RU-IN', 'ing'],
                    ['RU-SE', 'ost'], ['RU-TY', 'tva'],
                    ['RU-KC', 'krch'], ['RU-AD', 'adg'], ['RU-KAM', 'kmch'], ['RU-KL', 'klm'], ['RU-ALT', 'alt'], ['RU-MAG', 'mgd'],
                    ['RU-YEV', 'eao'], ['RU-CHU', 'chuk'], ['RU-NEN', 'nao']]

def save(source, name, directory, region_code = None):
    ru_region_save = pd.read_html(requests.get(source, headers=headers).text)    # <<--
    sleep(5)    # <<--
    ru_region_save = ru_region_save[0]
    if region_code != None:
        ru_region_save['region_code'] = region_code
    ru_region_save.to_csv(directory + name, index=False)

# whatever you need..    
directory = './'


save(source=query_dataset, name='now.csv', directory=directory)

HTTP Error 503 when trying to download data from the site

2 Answers