I tried to do web scraping with python and the output is empty. did i do something wrong?

user_14380702 0
Viewed 117

this is the code that i used, you can see that i copied the result page and tried to print it and the output is [ ]. I'm trying to learn web-scraping. so, i'm trying to web-scrape the name of the orphanage and copy inta csv file. but, i couldn't get the first phase. The "result_page" exists.

import os
os.system('cls')

import pandas as pd
import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.justdial.com/Chennai/Orphanages/nct-10344906')
soup = BeautifulSoup(page.content,'html.parser')
MainContent = soup.find_all(class_="result_page")
print(MainContent)
web-scraping beautifulsoup python-3.x python
1 Answers

If you are getting this error even though the page has result_page at the root, it indicates that the page has been blocked

[UPDATE1]

I tried this for about access or not :

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://www.justdial.com/Chennai/Orphanages/nct-10344906')
soup = bs(r.content,'html.parser')
print(soup)

and output:

<html><head>
<title>Access Denied</title>
</head><body>
<h1>Access Denied</h1>

You don't have permission to access "http://www.justdial.com/Chennai/Orphanages/nct-10344906" on this server.<p>
Reference #18.95a0de52.1603091762.1ae82063
</p></body>
</html>

[UPDATE2]

finally unblocked

code:

import requests
from bs4 import BeautifulSoup as bs

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}

r = requests.get('https://www.justdial.com/Chennai/Orphanages/nct-10344906',headers=headers).text
soup = bs(r,'html.parser')
soup = soup.find("div",{"class":"result_page"})
print(soup)

NOT: If you still get errors, make sure you are using the correct user-agent. go to google site press to F12 and Network, refresh the page and press a thing

enter image description here

user_126339710
Related