How to simulate a browser with Python requests

Viewed 63

I want to make a google search or enter some data on a website and then do the same with Python.

Basically doing an action on my browser and then doing it again with the same request data with python.

I had the idea of using an https traffic recorder (browser extension) but it's just too painful to just copy all the headers and etc.

I could use selenium but it's just way too slow and it's not too easy to also manipulate data that are being sent.

1 Answers

Anything which will involves entering some data on a website or clicking some buttons can't be done with http requests unless you are an experienced hacker with experience in manipulating and analyzing http response of your request. I recommend sticking with commonly practiced tools like selenium where it involves clicking and entering text etc.

However if you just want to get the google search results using python, you can add the user-agent header to achieve that.

import requests

url='https://www.google.com/search?q=hello+world'
headers = {"User-Agent": "Mozilla/5.0", "accept-language": "en-US,en"}
print(requests.get(url,headers=headers).status_code)
Related