Want to scrape a whole webpage which has the loadmore button with using post method(don't wanna use selenium) through beautifulsoup in python

user_18904011 0
Viewed 59

I wanna scrape some details from a main_url in which for each company it has another url for each company and get credentials of a each company from which there are some elements like company name,phone,fax,website..etc. I have write the code using beautifulsoup and requests and also got the credentials(but got only for 52 companies) .After that it giving an error of javascript cause there is loadmore button. I want to get all companies details by crossing that loadmore button too.Only wanna do with requests and beautifulsoup. Don't want to use selenium for it.I'll be glad and thankful for getting help. Here is my code till where I get good result but only having problem with load more button.

from time import time
from bs4 import BeautifulSoup
import urllib.request
import requests
import json
import time

parser = 'html.parser'  # or 'lxml' (preferred) or 'html5lib', if installed
resp = urllib.request.urlopen("https://www.arabiantalks.com/category/1/advertising-gift-articles")
soup = BeautifulSoup(resp, parser, from_encoding=resp.info().get_param('charset'))
cnt=0

for links in soup.find_all('a', href=True):
    url=(links['href'])

    # print(url)
    page=requests.get(url)
    # print(page).

    soup=BeautifulSoup(page.content,'html.parser')

    # print(soup)
    results=soup.find('div',class_='rdetails')
    # print(results)
   


    if results is not None:
        # print(f"company_results: {results.text}")
        Name=results.find('h1')
        if Name is not None:
            print(f"Comapny_name: {Name.text}")
        else:
            print(f"Comapny_name: Notfound")

        address=results.find(attrs={'itemprop':'address'})
        if address is not None:
            # address.text.replace(" Address :   ", "")
            # print(type(f"Comapny_address: {address.text}"),"this type")
            print((f"Comapny_address: {address.text[12:]}"))
        else:
            print(f"Comapny_address: Notfound")
        phone=results.find(attrs={'itemprop':'telephone'})
        if phone is not None:
            print(f"Comapny_phone: {phone.text[16:]}")
        else:
            print(f"Comapny_phone: Notfound")

        fax=results.find(attrs={'itemprop':'faxNumber'})
        if fax is not None:
            print(f"Comapny_fax: {fax.text[7:]}")
        else:
            print(f"Comapny_fax: Notfound")

        email=results.find(attrs={'itemprop':'email'})
        if email is not None:
            print(f"Comapny_email: {email.text[9:]}")
        else:
            print(f"Comapny_email: Notfound")

        website=results.find(attrs={'itemprop':'url'})
        if website is not None:
            print(f"Comapny_website: {website.text}")
            
        else:
            print(f"Comapny_website: Notfound")
            
        cnt += 1

        print(cnt) 
     
        print("="*100)

If anyone know the ideology could you please copy these code and do required modifications and post as reply again.So I would know where to change.I tried from many articles, but still struggling.This is first answer the final link I tried.Please help me with this issue I'm pretty new in web scraping. Thanks in advance...

web-scraping beautifulsoup python
1 Answers

This question is (almost) a duplicate of this one: Scraping a website that has a "Load more" button doesn't return info of newly loaded items with Beautiful Soup and Selenium

The difference is - on this instance, the ajax response is not JSON, but HTML. You need to inspect the Network tab in browser's Dev tools, to see what network calls are being made.

The following code will access the ajax endpoint, pull all the data available, get all companies profile urls, scrape name, address, phone, fax, email, website and save everything into a csv file:

from bs4 import BeautifulSoup
import requests
import pandas as pd

item_list = []
counter = 20

s = requests.Session()

r = s.get('https://www.arabiantalks.com/category/1/advertising-gift-articles')
soup = BeautifulSoup(r.text, 'html.parser')
items = soup.find_all('a', {'itemprop': 'item'})

for item in items:
    item_list.append(item.get('href'))

while True:
    payload = {
        'start': counter,
        'cat': 1
    }
    r = s.post('https://www.arabiantalks.com/ajax/load_content', data=payload)
    if len(r.text) < 25:
        break
    soup = BeautifulSoup(r.text, 'html.parser')
    items = soup.find_all('a')
    for item in items:
        item_list.append(item.get('href'))
    counter = counter + 12
print('Total items:', len(set(item_list)))
full_comp_list = []
for x in item_list:
    r = s.get(x)
    soup = BeautifulSoup(r.text, 'html.parser')
    c_details_card = soup.select_one('div.rdetails')
    try:
        c_name = c_details_card.select_one('#hcname').text.strip()
    except Exception as e:
        c_name = 'Name unknown'
    try:
        c_address = c_details_card.find('h3', {'itemprop': 'address'}).text.strip()
    except Exception as e:
        c_address = 'Address unknown'
    try:
        c_phone = c_details_card.find('h3', {'itemprop': 'telephone'}).text.strip()
    except Exception as e:
        c_phone = 'Phone unknown'
    try:
        c_fax = c_details_card.find('h3', {'itemprop': 'faxNumber'}).text.strip()
    except Exception as e:
        c_fax = 'Fax unknown'
    try:
        c_email = c_details_card.find('h3', {'itemprop': 'email'}).text.strip()
    except Exception as e:
        c_email = 'Email unknown'
    try:
        c_website = c_details_card.find('a').get('href')
    except Exception as e:
        c_website = 'Website unknown'
    full_comp_list.append((c_name, c_address, c_phone, c_fax, c_email, c_website))
    print('Done', c_name)

full_df = pd.DataFrame(list(set(full_comp_list)), columns = ['Name', 'Address', 'Phone', 'Fax', 'Email', 'Website'])
full_df.to_csv('full_arabian_advertising_companies.csv')
full_df

It will also print out in terminal, to give you a sense of what it's doing:

Total items: 122
Done Ash & Sims Advertising LLC
Done Strings International Advertising LLC
Done Zaabeel Advertising LLC
Done Crystal Arc Factory LLC
Done Zone Group
Done Business Link General Trading
[....]

Name    Address Phone   Fax Email   Website
0   Ash & Sims Advertising LLC  Address : P.O.Box 50391,\nDubai - United Arab Emirates  Phone Number : +971-4-8851366 , +9714 8851366   Fax : +971-4-8852499    E-mail : sales@ashandsims.com   http://www.ashandsims.com
1   Strings International Advertising LLC   Address : P O BOX 117617\n57, Al Kawakeb Property, Al Quoz\nDubai, U.A.E    Phone Number : +971-4-3386567 , +971502503591   Fax : +971-4-3386569    E-mail : vinod@stringsinternational.org http://www.stringsinternational.org
2   Zaabeel Advertising LLC Address : Al Khabaisi,  Phone Number : +971-4-2598444   Fax : +971-4-2598448    E-mail : info@zaabeeladv.com    http://www.zaabeeladv.com
3   Crystal Arc Factory LLC Address : Dubai - P.O. Box 72282\nAl Quoz, Interchange 3, Al Manara,    Phone Number : +971-4-3479191 , +971 4 3479191, Fax : +971-4-3475535    E-mail : info@crystalarc.net    http://www.crystalarc.net
4   Zone Group  Address : Al Khalidiya opp to Rak Bank, Kamala Tower,\nOffice no.1401, PO Box 129297, Abu Dhabi, UAE    Phone Number : +97126339004 Fax : +97126339005  E-mail : info@zonegroupuae.ae   http://www.zonegroupuae.ae
[....]
user_194751850
Related