Can't able to scrape the airtel website plans with python BeautifulSoup

Viewed 35
from bs4 import BeautifulSoup
import requests

req = requests.get("https://www.airtel.in/myplan-infinity/")

  
soup = BeautifulSoup(req.content, 'html.parser')
  
#finding the div with the id
div_bs4 = soup.find('div')
print(div_bs4)

What should I do to scrape the recharge plans of the page?

1 Answers

you should use content when you need to get media data (like pictures, videos, etc)
if you need to get non-media data you've to use text instead of content

make the soup like this: `soup = BeautifulSoup(req.text, 'lxml')

make sure that this like div_bs4 = soup.find('div') finds the exact div you need (cuz it will just get the first div in html)

this line print(div_bs4) won't give you needed data
you'd better use this print(div_bs4.text)

Related