How to Web Scrape the Navigation Bar of a website while maintaining hierarchy of contents

Rovsh 0
Viewed 53

I am trying to web scrape the contents of a website's navigation bar details but at the same time I also want to maintain the hierarchy say in a dictionary, for e.g.,

https://www.tyremarket.com/Car-Tyres has the following contents in the dropdown list... enter image description here

I used BeautifulSoup to get the scrape the contents

from bs4 import BeautifulSoup as bs
import requests
url = 'https://www.tyremarket.com/Car-Tyres'
html_text = requests.get(url).text
soup = bs(html_text, 'html.parser')
print (soup.nav)

for string in soup.nav.stripped_strings:
    print(string.strip('-'))

OUTPUT:

Car Tyres
Two-Wheeler Tyres
SCV Tyres
Services
Lubricants
3D Mats
Offers
Tyre Mantra
Advisory
Featured
News
Videos
Tyre Brand History
Road Tales
Cars and Bikes
Journey
Trendy
Auto Geeks
Videos
Seller Solutions
Dealer Registration

In this porcess the hierarchial information is lost. What I would like to have is convert this info to a dictionary where keys are the main headers of the navigation bar and if there are any drop down contents they would be present as a list.

Example -

{'Tyre Mantra':['Advisory','Featured','News','Videos']}
beautifulsoup web-scraping selenium python
1 Answers

Try to iterate the first levels and extract texts of each and create your dict structure:

d = {}

for e in soup.select('nav ul li'):
    s = list(e.stripped_strings)
    d.update({s[0]: s[1:]})

Example

from bs4 import BeautifulSoup as bs
import requests
url = 'https://www.tyremarket.com/Car-Tyres'
html_text = requests.get(url).text
soup = bs(html_text, 'html.parser')
d = {}

for e in soup.select('nav ul li'):
    s = list(e.stripped_strings)
    d.update({s[0]: s[1:]})

print(d)

Output:

{'Car Tyres': [],
 'Two-Wheeler Tyres': [],
 'SCV Tyres': [],
 'Services': [],
 'Lubricants': [],
 '3D Mats': [],
 'Offers': [],
 'Tyre Mantra': ['Advisory',
  'Featured',
  'News',
  'Videos',
  'Tyre Brand History'],
 'Advisory': [],
 'Featured': [],
 'News': [],
 'Videos': [],
 'Tyre Brand History': [],
 'Road Tales': ['Cars and Bikes', 'Journey', 'Trendy', 'Auto Geeks', 'Videos'],
 'Cars and Bikes': [],
 'Journey': [],
 'Trendy': [],
 'Auto Geeks': [],
 'Seller Solutions': ['Dealer Registration'],
 'Dealer Registration': []}
HedgeHog0
Related