ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

We are trying to scrape a large number of company numbers from companies house using the Companies House API but keep getting these below errors after running for a while. We are working with a large number of requests (650000) and the rate limit is 600 in 5 mins. We have addressed this in our code by sleeping for 5 mins after every 599 requests.

After returning 11,815 requests we get the following errors:

ConnectionResetError: [Errno 54] Connection reset by peer

ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

Our code:

import requests
import pandas as pd
import json
import time
from tqdm import tqdm
from datetime import datetime


# director_ids from director dump first scrape
df = pd.read_csv("dir_info.csv", encoding= "utf-8")
dir_ids_0 = df['links.officer.appointments'].tolist()    

#API KEY    
api_key = "XXXXXXXXXXXXXX"

#URL 
base_url = "https://api.company-information.service.gov.uk/"

request_number = 0

co_list_a =[]
dir_num_error ={}

for item in tqdm(dir_ids_0):
    if request_number > 599:
        print("sleeping")
        time.sleep(300)
        request_number = 0
    else: pass 
    response = requests.get(f"{base_url}{item}",auth=(api_key,''))
    request_number = request_number + 1
    dir_num_error[item] = {}
    dir_num_error[item]['status_code'] = response.status_code
    dir_num_error[item]['timestamp'] = str(datetime.now())
    dir_num_error[item]['request_number'] = request_number 
    if response.status_code == 429:
        print("429_sleeping")
        time.sleep(300)
        request_number = 0
        continue
    else: pass
    if response.status_code != 200:
        continue
    else: pass
   
    json_search_result = response.text
    data = json.JSONDecoder().decode(json_search_result)
    for item in data['items']:
          co_list_a.append(item['links']['company'])

As you can see we created a status code dictionary in the code so that we could monitor any status code errors we may get. From the last attempt, we did not find any evidence that the problem was a request problem (as status codes were fine). Any advice on this would be very helpful.