Can someone please explain what happens if I don't specify my user agent in requests.get(url)?
If I specify, it'd be like the following:
user_agent = {'User-agent': 'Mozilla/5.0'}
requests.get(url, headers=user_agent)
And if I don't, it'd simply be
requests.get(url)
What difference does it make? And what's the default for user agent if I don't specify?
Thanks in advance!
The default Python Requests user agent will be used. In my case (Python 3.9.1) the default user agent is python-requests/2.25.0.
Some websites block access from non web browser User Agents to prevent scraping.
If you are connected to a webpage, your browser sends a user agent to the website. Not specifying a user agent has consequences for websites that block webscraping, as they will detect you are using an automated browser. So by using a user agent, you will mimick a normal browser, thus less likely to be blocked by the website.
The User-Agent header when typically sent by a web browser or other http client, does not have ANY defined meaning. It simply tells the server something about the client, and the server can do whatever it wants with that information.
Normally this would be used to alter the response to maintain compatibility with different clients (browsers). For example, a web application might respond with a simplified site if it detects a mobile browser via the User-Agent string.
But there are no hard and fast rules here. The server may just ignore this header completely.
In other cases you may find applications, especially older ones, that were coded to rely on features that were available only in specific browsers, and that will not return a useful response if they don't recognize the client as a browser that they like. In that case, it's useful to set the User-Agent header to that of a well-supported browser that the application is coded to handle. It's not how things SHOULD work, but sometimes that's necessary.
By default, requests sends a user-agent header which identifies the version of the requests package you are using, the python version and the operating system.
Most web applications aren't written to recognize python-requests as a browser so, if they are coded to be particular about the browser used to access them, they may not return a good response, in which case you'll need to fake the header to make it look like you are using a more common browser that the server side application recognizes.
According to the RFC7231:
The "User-Agent" header field contains information about the user agent originating the request, which is often used by servers to help identify the scope of reported interoperability problems, to work around or tailor responses to avoid particular user agent limitations, and for analytics regarding browser or operating system use. A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.
From my point of view, this value is not very useful for you if you are web scraping.
Some web site can block your requests if your program doesn’t look like a web browser. It cost nothing to use “Mozilla/5.0”.