python stdin and pass to requests.get

user_11014335 0
Viewed 39

I want to pass a stdin [file which have lines] to requests.get

For example,

I want to check status code of below websites.

http://google.com
http://bing.com
http://twitter.com

Below is a correct code for checking status code of a single website. Which command is "test.py --url http://google.com"

Now I want to check status code of multiple websites from an input file. E.g: "test.py --website url.txt"

def run(arguments):
    if arguments.url is None or arguments.url.strip() == '':
        target = get_input("Enter URL : ")
    else:
        target = (arguments.url)


    var = requests.get(target, headers={"Referer":"test"}, verify=False)
    print(var.status_code, "ok")

Thanks

python
2 Answers

It's easier to just pass the file in through stdin with a command like this: test.py < url.txt. You can then use sys.stdin without having to worry about handling files. Here's an example:

import sys

def run():
    for url in sys.stdin:
        if url is None or url.strip() == '':
            continue

        var = requests.get(url, headers={"Referer":"test"}, verify=False)
        print(var.status_code, "ok")
user_133765110 
    with open(path_for_urls, 'r') as f:
        for url_line in f.readlines():
            var = requests.get(url_line, headers={"Referer": "test"}, verify=False)

you can get the same code piece to get the file path. Then process the each line of the file with the example above. If your file is too big and having memory issues (like millions of URLs to check) then reading line by line, instead of readlines() would be better.

user_45879310
Related