I have seen that there are many approarches for scraping data from google scholar out there. I tried to put together my own code, however I cannot get access to google scholar using free proxies. I am interested in understanding why that is (and, secondarily, what to change). Below is my code. I know it is not the most elegant one, its my first try at data scraping...
This is a list of proxies I got from "https://free-proxy-list.net/" and I did test if they worked by accessing "http://icanhazip.com" with them.
live_proxies = ['193.122.71.184:3128', '185.76.10.133:8081', '169.57.1.85:8123', '165.154.235.76:80', '165.154.235.156:80']
Then I made the urls I want to scrape, and tried to get the content of the pages with one randome proxy
search_terms = ['Acanthizidae', 'mammalia']
for i in range(len(search_terms)):
url = 'https://scholar.google.de/scholar?hl=en&as_sdt=0%2C5&q={}&btnG='.format(search_terms[i])
session = requests.Session()
proxy = random.choice(live_proxies)
session.proxies = {"http": proxy , "https": proxy}
ua = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
output = s.get(url, headers=ua, timeout=1.5).text
However, I get:
requests.exceptions.ProxyError: HTTPSConnectionPool(host='scholar.google.de', port=443): Max retries exceeded with url: /scholar?hl=en&as_sdt=0%2C5&q=Acanthizidae&btnG= (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request')))
Like I said, I did test the proxies before with a different site. What is the problem?