Script fails to extract certain fields from a webpage using requests

user_13354003 0
Viewed 45

I'm trying to scrape title, website and email from this webpage. The content available in there are heavily dynamic. Although requests module can't handle javascript heavy content, there are always alternatives to grab the same using the very library, as in using any external link retrieved from dev tools. However, I just can't find the right way to do the trick.

I've tried with:

import requests
from bs4 import BeautifulSoup

link = 'https://www.firmy.cz/detail/13153157-azajola-stare-mesto-nova-seninka.html'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    title = soup.select_one("h1").text
    website = soup.select_one("a.companyUrl").text
    email = soup.select_one("a.companyMail").text
    print(title,website,email)

When I run the script above, it throws AttributeError.

Output that I'm after:

AZAJOLA, s.r.o., www.chatanovaseninka.cz, info@chatanovaseninka.cz

PS I don't wish to go for any browser simulator like selenium.

python-requests web-scraping python-3.x python
1 Answers

The site uses FastRPC protocol (which is additionally encoded with Base64). You can install PyFRPC module from https://pypi.org/project/pyfrpc/ to encode/decode the messages:

import re
import pyfrpc  # <-- install from https://pypi.org/project/pyfrpc/
import base64
from pprint import pprint

url = (
    "https://www.firmy.cz/detail/13153157-azajola-stare-mesto-nova-seninka.html"
)
api_url = "https://www.firmy.cz/RPC2/"
id_ = int(re.search(r"detail/(\d+)", url).group(1))

headers = {
    "Accept": "application/x-base64-frpc",
    "Content-Type": "application/x-base64-frpc",
}
c = pyfrpc.FrpcCall(
    name="system.multicall",
    args=[
        [
            {
                "methodName": "detail.getDetail",
                "params": [
                    {"version": "1.0"},
                    id_,
                    {"searchInCategory": False, "deliveryNetwork": ""},
                ],
            },
            {
                "methodName": "review.listReviews",
                "params": [{"version": "1.0"}, id_, 0, 3],
            },
        ]
    ],
)
msg_to_send = base64.b64encode(pyfrpc.encode(c, 0x0201))

r = requests.post(api_url, headers=headers, data=msg_to_send)
response = pyfrpc.decode(base64.b64decode(r.text))

# uncomment to see all data:
# pprint(response.data)

print(response.data[0]["result"]["title_web"])
print(response.data[0]["result"]["email"])
print(response.data[0]["result"]["url"].split("#")[0])

Prints:

AZAJOLA, s.r.o.
info@chatanovaseninka.cz
http://www.chatanovaseninka.cz
user_100359850
Related