With a small dataframe to iterrate through to obtain the status of a URL, it took 49 minutes to complete. Out of the 1300 records, 128 resulted in an exception due to connection errors or security errors. I wonder if there is a more efficient and quicker way to iterrate through the records and reduce the time.
opendiners = []
df_link = pd.read_csv('diners.csv')
df_link=df_link.dropna(subset=['link'])
df_link['link'] = df_link['link'].str.strip()
df_link = df_link[df_link.link != '']
for link in df_link.iterrows():
try:
url = link[1]['link']
title = link[1]['title']
address = link[1]['address']
phone = link[1]['phone']
requests.adapters.DEFAULT_RETRIES = 5
response = requests.get(url, headers = hdr, timeout=None, allow_redirects=False)
if response.status_code >= 0:
diners = {
'answer': response,
'title': title,
'address': address,
'phone': phone,
'website': url
}
opendiners.append(diners)
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
df = pd.DataFrame(opendiners)
| title | link | phone |
|---|---|---|
| Pie Town Tacos | http://pietowntacos.com/ | (615) 750-2628 |