beautifulsoup not returning complete html of the page in python

user_19567184 0
Viewed 74

I have taken help to extract data from the package BeautifulSoup, but unfortunately, it does not give me the output of the complete HTM file.

   from bs4 import BeautifulSoup
   import requests

   url = 'https://www.digikala.com/search/category-mobile-phone/?page=1'
   response = requests.get(url)
   print(response.text)
   soup = BeautifulSoup(response.content,"lxml")
   m= soup.find_all("h2", {'class':'ellipsis-2 text-body2-strong 
    color-700'})
   print(m)
web-scraping beautifulsoup python
1 Answers

This page is loading data from an API. You can see this by inspecting Network tab in Chrome Dev Tools, reloading the page.

The following code will return a json with all items from page 1:

from bs4 import BeautifulSoup
import requests

url = 'https://api.digikala.com/v1/categories/mobile-phone/search/?page=1'
response = requests.get(url).json()
print(response)
user_194751850
Related