Improve Speed for Requests.Get Iterrating Through 1300 Records

Viewed 40

With a small dataframe to iterrate through to obtain the status of a URL, it took 49 minutes to complete. Out of the 1300 records, 128 resulted in an exception due to connection errors or security errors. I wonder if there is a more efficient and quicker way to iterrate through the records and reduce the time.

opendiners = []
df_link = pd.read_csv('diners.csv')

df_link=df_link.dropna(subset=['link'])
df_link['link'] = df_link['link'].str.strip()
df_link = df_link[df_link.link != '']

for link in df_link.iterrows():
    try:
      url = link[1]['link']
      title = link[1]['title']
      address = link[1]['address']
      phone = link[1]['phone']
      requests.adapters.DEFAULT_RETRIES = 5
      response = requests.get(url, headers = hdr, timeout=None, allow_redirects=False)
    
      if response.status_code >= 0:
          diners = {
          'answer': response,
          'title': title,
          'address': address,
          'phone': phone,
          'website': url
          }
          opendiners.append(diners)
    except requests.exceptions.HTTPError as errh:
        print ("Http Error:",errh)
    except requests.exceptions.ConnectionError as errc:      
        print ("Error Connecting:",errc)
    except requests.exceptions.Timeout as errt:
        print ("Timeout Error:",errt)
    except requests.exceptions.RequestException as err:
        print ("OOps: Something Else",err)
        
df = pd.DataFrame(opendiners)
title link phone
Pie Town Tacos http://pietowntacos.com/ (615) 750-2628
0 Answers
Related