Beautifulsoup - scrape webpage - dynamically loading page

Viewed 3542

i want to scrape a webpage : https://www.justdial.com/Mumbai/Dairy-Product-Retailers-in-Thane/nct-10152687

i need a data of all store name, tel- num and their address

But i can only do it upto 10 cause to load other items you need to scroll the webpage

my code :

import requests
import bs4

crawl_url = requests.get('https://www.justdial.com/Mumbai/Dairy-Product-
Retailers-in-Thane/nct-10152687', headers={'User-Agent': 'Mozilla/5.0'})
crawl_url.raise_for_status()


soup = bs4.BeautifulSoup(crawl_url.text, 'lxml')

for elems in soup.find_all('span', class_="jcn"):
    select_a = elems.select('a')
    for links in select_a:
        href = links.get('href')
        res = requests.get(href, headers={'User-Agent': 'Mozilla/5.0'})

        xsoup = bs4.BeautifulSoup(res.text, 'lxml')

        Name = xsoup.select('.fn')
        tel = xsoup.select('.tel')
        add = xsoup.select('.adrstxtr')
        a = Name[0]
        b = tel[0]
        c = add[0]
        print(a.getText())
        print("--"*10)
        print(b.getText())
        print("--"*10)
        print(c.getText())
        print("=="*25)

When We Scroll Down the Page other Items Load up So i Want to Know How can get any numbers of Data/Items I want

I tried this

But didn't quiet understood Well, and also i did not got that POST method :/

If need more Info Tell me

1 Answers
Related