So i have been building a Django App using Beautiful soup to scrape a website and it was working perfectly in development. However, then deployed the web app to a linux server and ran it and that is when the the request was blocked and I got this error
You don't have permission to access "http://www.xxxxxx.com/en-ca?" on this server.<p>
Reference #xxxxxxxxxxxxxx
I tried using Mozilla and Chrome headers but that still did not work.
hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Saf$
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Referer': 'https://cssspritegenerator.com',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'en-US,en;q=0.8',
'Connection': 'keep-alive'}
req = get(url, headers=hdr)
soup = BeautifulSoup(req.content,'html.parser')
print(soup)
I also tried another solution I saw online that uses a particular request like this
class AppURLopener(urllib.request.FancyURLopener):
version = "Mozilla/5.0"
def scrape():
opener = AppURLopener()
response = opener.open(url)
print('response: ', response)
which gives me the response
response: <addinfourl at 139782695145200 whose fp = <_io.BufferedReader name=-1>>
which is something I do not understand
So I just needed some help as to what this response is and how can i use it?
Furthermore, should I look into using something like Selenium instead of beautifulSoup as it could be a better option to access the site undetected?
Plus, since the request is only blocked on the linux server in deployment should I try a different means of deployment such as heroku?