Can't scrape a graphql page using requests

user_7180194 0
Viewed 704

I'm trying to scrape firm names and their corresponding links from a web-page using requests module.

Although the content is heavily dynamic, I could notice that they are available within the braces next to window.props.

So, I thought to scoop out that portion and process it using json, but I see \u0022 characters around instead of quotes ". This is what I meant:

{\u0022firms\u0022: [{\u0022index\u0022: 1, \u0022slug\u0022: \u0022zjjz\u002Datelier\u0022, \u0022name\u0022:

I've tried with:

import re
import json
import requests
from bs4 import BeautifulSoup

link = 'https://architizer.com/firms/'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    r = s.get(link)
    items = re.findall(r'window.props[^"]+(.*?);',r.text)[0].strip('"').replace('\u0022', '\'')
    print(items)

How can I scrape the name and link of different firms traversing multiple pages from that web-page using requests?

graphql web-scraping beautifulsoup python-3.x python
1 Answers

Well, that was a fun one.

You're dealing with a page powered by GraphQL, so you have to mimic the request correctly.

Also, they want you to send a Referer Header along with a csfr token. That can be easily scooped out of the initial HTML and reused in subsequent requests.

Here's my take on this:

import time

import requests
from bs4 import BeautifulSoup

link = 'https://architizer.com/firms/'
query = """{ allFirmsWithProjects( first: 6, after: "6", firmType: "Architecture / Design Firm", firmName: "All Firm Names", projectType: "All Project Types", projectLocation: "All Project Locations", firmLocation: "All Firm Locations", orderBy: "recently-featured", affiliationSlug: "", ) { firms: edges { cursor node { index id: firmId slug: firmSlug name: firmName projectsCount: firmProjectsCount lastProjectDate: firmLastProjectDate media: firmLogoUrl projects { edges { node { slug: slug media: heroUrl mediaId: heroId isHiddenFromListings } } } } } pageInfo { hasNextPage endCursor } totalCount } }"""


def query_graphql(page_number: int = 6) -> dict:
    q = query.replace(f'after: "6"', f'after: "{str(page_number)}"')
    return s.post(
        "https://architizer.com/api/v3.0/graphql",
        json={"query": q},
    ).json()


def has_next_page(graphql_response: dict) -> bool:
    return graphql_response["data"]["allFirmsWithProjects"]["pageInfo"]["hasNextPage"]


def get_next_page(graphql_response: dict) -> int:
    return graphql_response["data"]["allFirmsWithProjects"]["pageInfo"]["endCursor"]


def get_firms_data(graphql_response: dict) -> list:
    return graphql_response["data"]["allFirmsWithProjects"]["firms"]


def parse_firms_data(firms: list) -> str:
    return "\n".join(firm["node"]["name"] for firm in firms)


def wait_a_bit(wait_for: float = 1.5):
    time.sleep(wait_for)


with requests.Session() as s:
    s.headers["user-agent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36"
    s.headers["referer"] = "https://architizer.com/firms/"

    csrf_token = BeautifulSoup(
        s.get(link).text, "html.parser"
    ).find("input", {"name": "csrfmiddlewaretoken"})["value"]

    s.headers.update({"x-csrftoken": csrf_token})

    response = query_graphql()
    while True:
        if not has_next_page(response):
            break
        print(parse_firms_data(get_firms_data(response)))
        wait_a_bit()
        response = query_graphql(get_next_page(response))

This should output, just for the sake of the example, firms' names:

Brooks + Scarpa Architects
Studio Saxe
NiMa Design
Best Practice Architecture
Gensler
Inca Hernandez
kaa studio
Taller Sintesis
Coryn Kempster and Julia Jamrozik
Franklin Azzi Architecture
Wittman Estes
Masfernandez Arquitectos
MATIAS LOPEZ LLOVET
SRG Partnership, Inc.
GANA Arquitectura
Meyer & Associates Architects, Urban Designers
Steyn Studio
BGLA architecture | urban design

and so on ...
user_61067910
Related