Output file of for loop with no values

user_16390642 0
Viewed 45

I have the below script to go through list of URLs, and return the response.status_code of the server.

The output is working fine, however I am trying to save the output into pandas dataframe.. which turns out to be empty for some reason.

Below is the code:

import requests
import webbrowser
import pandas as pd
import csv 
import string
from time import sleep as time


df = pd.read_csv("status_url.csv")
#df.info()
#df.head()
  
df2 = pd.DataFrame() # creating new df to save the output of the loop below 
df2['url_2'] = ''
df2['result'] = ''

def check(filename):
    with open(filename, 'r') as f_in:
        reader = csv.DictReader(f_in)
        for row in df['Url']:
            Url = df['Url']
            try:
                response = requests.get(row,timeout=3)
                if(response.status_code) == 200:
                    print(row, response.status_code) 
                    row = df2['url_2'] 
                    response.status_code = df2['result'] 
            except requests.exceptions.HTTPError as errh:
                row = df2['url_2'] 
                df2['result'] = 'HTTP Error'             
            except requests.exceptions.ConnectionError as errc:
                print (row,"Error Connecting: Failed to establish a new connection") # this print line works correctly 
                row = df2['url_2'] 
                df2['result'] = "Error Connecting: Failed to establish a new connection"
            except requests.exceptions.Timeout as errt:
                row = df2['url_2'] 
                df2['result'] = "Timeout Error"
            except requests.exceptions.RequestException as err:
                row = df2['url_2'] 
                df2['result'] = 'OOps: Something Else'

                         
df2.to_csv('URLs_Output.csv', sep=',', index=False) # this file is empty while it should not.        
  
                      
filename = 'status_url.csv'
check(filename)

Any ideas how to save the output to pandas as csv file? where it has two columns(row, which is the url) and the result(200, or error, etc)

Thank you

python-requests pandas python-3.x
2 Answers

There's a bit going on here and no data for me to use, but I can identify a few issues with the code sample that need some modification.

  • First of all, you are correct to not modify the dataframe you are iterating over. While that can work, it also can lead to difficult, hard to understand errors, and should in general be avoided.
  • However, you can avoid doing that by creating a copy of the dataframe, not an empty dataframe.

Try defining your dataframes like this:

df = pd.read_csv("status_url.csv")

  
df2 = df.copy() # create copy of DF to save output
df2['result'] = 'NOT CHECKED' # create column for output
# I put this value so it'll be clear if we don't update it

  • Nothing is being done in your code with the csv.reader object. Get rid of it.

  • Iterate through df, store values to df2.

# Iterate through df urls
# we can use the idx because df2 was created as a copy of df
for idx,url in df['Url'].iteritems():
    try:
        response = requests.get(url,timeout=3)
        if(response.status_code) == 200:
            print(url, response.status_code) 
            # Store output to df2
            df2.at[idx,'result'] = str(response.status_code)
    except requests.exceptions.HTTPError as errh:
        df2.at[idx,'result'] = 'HTTP Error'             
    except requests.exceptions.ConnectionError as errc:
        print (row,"Error Connecting: Failed to establish a new connection") # this print line works correctly 
        df2.at[idx,'result'] = "Error Connecting: Failed to establish a new connection"
    except requests.exceptions.Timeout as errt:
        df2.at[idx,'result'] = "Timeout Error"
    except requests.exceptions.RequestException as err:
        df2.at[idx,'result'] = 'OOps: Something Else'

Without a reproducible data sample I can't test my code, but I think it will work.

user_62324830

So this is the answer ^^

def check(filename):
    with open(filename, 'r') as f_in:
        reader = csv.DictReader(f_in)
        with open('test_output.csv', 'a', encoding="utf-8") as csv_file:
            writer = csv.writer(csv_file, delimiter='>')
            for row in df['Url']:
                Url = df['Url']
                try:
                    response = requests.get(row,timeout=3)
                    if(response.status_code) == 200:
                        print(row,"*", response.status_code) 
                        #row = df2['url_2'] 
                        result ='200'
                        writer.writerow([row, result])
                except requests.exceptions.HTTPError as errh:
                    result = 'HTTP Error' 
                    writer.writerow([row, result])
                except requests.exceptions.ConnectionError as errc:
                    #print (row,"*Error Connecting: Failed to establish a new connection") # this print line works correctly 
                    result = "*Error Connecting: Failed to establish a new connection"
                    writer.writerow([row, result])
                except requests.exceptions.Timeout as errt:
                    result = "*Timeout Error"
                    writer.writerow([row, result])
                except requests.exceptions.RequestException as err:
                    result = '*OOps: Something Else'
                    writer.writerow([row, result])
user_163906420
Related