Scraping country specific product pages with BeautifulSoup

user_13984877 0
Viewed 70

I'm successfully scraping the below website using BeautifulSoup, however the list of products on the page changes based on the location of the user. How can I include a location tag/cookie in order to only scrape the products relevant to my country?

https://www.themacallan.com/en/shop

web-scraping beautifulsoup python
1 Answers

The data is loaded from external URL via Ajax. You can use this example how to change the location and load the product feed:

import json
import requests

headers = {"Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09"}

# los angeles
url = "https://api.edrington.com/consumer/v1/macallan/en/US/Product/location/91605"

data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for item in data["items"]:
    print("{:<50} {}".format(item["title"], item["availableForSale"]))

Prints:

The Macallan 1824 Series Scotch No. 6              True
The Macallan Sherry Oak 30 Years Old               True
The Macallan Estate                                True
The Macallan Edition No. 5                         True
The Macallan Sherry Oak 25 Years Old               True
The Macallan Reflexion                             True
The Macallan Double Cask 18 Years Old              True

For zip code 10001 - New York use url = "https://api.edrington.com/consumer/v1/macallan/en/US/Product/location/10001":

The Macallan Sherry Oak 12 Years Old               True
The Macallan Rare Cask                             True
The Macallan Sherry Oak 18 Years Old               True
The Macallan Sherry Oak 30 Years Old               True
The Macallan Edition No. 5                         True
The Macallan Classic Cut 2019 Edition              True
The Macallan Edition No. 4                         True
The Macallan Sherry Oak 25 Years Old               True
The Macallan Reflexion                             True
The Macallan Double Cask 12 Years Old              True
The Macallan Double Cask 15 Years Old              True
The Macallan Double Cask 18 Years Old              True
The Macallan Edition No. 6                         True

EDIT: For Great Britain and list of URLs:

import json
import requests
from bs4 import BeautifulSoup


headers = {"Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09"}

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"

data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

available_ids = {i["productId"] for i in data["items"]}

soup = BeautifulSoup(
    requests.get("https://www.themacallan.com/en/shop").content, "html.parser"
)

for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print("https://www.themacallan.com" + i.a["href"])

Prints:

https://www.themacallan.com/en/double-cask-18-years-old
https://www.themacallan.com/en/whisky/single-malts/rare-cask/rare-cask-2020-release
https://www.themacallan.com/en/whisky/single-malts/sherry-oak-25-years-old-2019-release
https://www.themacallan.com/en/the-macallan-lumina-whisky
https://www.themacallan.com/en/the-macallan-fine-oak-10-years-old
https://www.themacallan.com/en/the-macallan-no-6
https://www.themacallan.com/en/the-macallan-quest-whisky
https://www.themacallan.com/en/the-macallan-terra-whisky
https://www.themacallan.com/en/triple-cask-matured-18-years-old
https://www.themacallan.com/en/the-macallan-lalique-tumbler
https://www.themacallan.com/en/the-macallan-rocks-glass-tumblers

NOTE: All products are loaded inside the page. You need only to filter them based on available_ids.

EDIT 2: To get "uncached" page:

import json
import requests
from bs4 import BeautifulSoup

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"


headers = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "cache-control": "no-cache",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "TE": "Trailers",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

options_headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Access-Control-Request-Headers": "cache-control,ocp-apim-subscription-key",
    "Access-Control-Request-Method": "GET",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

with requests.Session() as s:
    s.options(url, headers=options_headers).text
    data = s.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

available_ids = {i["productId"] for i in data["items"]}

soup = BeautifulSoup(
    requests.get("https://www.themacallan.com/en/shop").content, "html.parser"
)

for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print("https://www.themacallan.com" + i.a["href"])

Prints:

https://www.themacallan.com/en/double-cask-12-years-old
https://www.themacallan.com/en/sherry-oak-18-years-old
https://www.themacallan.com/en/whisky/single-malts/sherry-oak/sherry-oak-12-years-old
https://www.themacallan.com/en/sherry-oak-25-years-old
https://www.themacallan.com/en/whisky/single-malts/macallan-estate
https://www.themacallan.com/en/estate3
https://www.themacallan.com/en/triple-cask-matured-12-years-old
https://www.themacallan.com/en/distilling-scotland
https://www.themacallan.com/en/the-macallan-chevron-highball
https://www.themacallan.com/en/chevron-tumblers
https://www.themacallan.com/en/the-macallan-chevron-water-jug
https://www.themacallan.com/en/the-macallan-hip-flask
https://www.themacallan.com/en/the-macallan-ice-ball-maker
https://www.themacallan.com/en/the-macallan-in-lalique-glass
https://www.themacallan.com/en/the-macallan-polo-shirt

EDIT 3: Now with pagination:

import json
import requests
from bs4 import BeautifulSoup

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"


headers = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "cache-control": "no-cache",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "TE": "Trailers",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

options_headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Access-Control-Request-Headers": "cache-control,ocp-apim-subscription-key",
    "Access-Control-Request-Method": "GET",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

available_ids = set()
with requests.Session() as s:

    u = url
    while True:
        s.options(u, headers=options_headers).text
        data = s.get(u, headers=headers).json()

        for i in data["items"]:
            available_ids.add(i["productId"])

        if not data["hasNextPage"]:
            break

        u = url + "?after=" + data["lastCursor"]

    soup = BeautifulSoup(
        s.get("https://www.themacallan.com/en/shop").content,
        "html.parser",
    )

idx = 1
for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print(idx, "https://www.themacallan.com" + i.a["href"])
        idx += 1

Prints:

1 https://www.themacallan.com/en/double-cask-12-years-old
2 https://www.themacallan.com/en/double-cask-18-years-old
3 https://www.themacallan.com/en/whisky/single-malts/rare-cask/rare-cask-2020-release
4 https://www.themacallan.com/en/sherry-oak-18-years-old
5 https://www.themacallan.com/en/whisky/single-malts/sherry-oak/sherry-oak-12-years-old
6 https://www.themacallan.com/en/sherry-oak-25-years-old
7 https://www.themacallan.com/en/whisky/single-malts/sherry-oak-25-years-old-2019-release
8 https://www.themacallan.com/en/the-macallan-lumina-whisky
9 https://www.themacallan.com/en/whisky/single-malts/macallan-estate
10 https://www.themacallan.com/en/estate3
11 https://www.themacallan.com/en/the-macallan-fine-oak-10-years-old
12 https://www.themacallan.com/en/the-macallan-no-6
13 https://www.themacallan.com/en/the-macallan-quest-whisky
14 https://www.themacallan.com/en/the-macallan-terra-whisky
15 https://www.themacallan.com/en/triple-cask-matured-12-years-old
16 https://www.themacallan.com/en/triple-cask-matured-18-years-old
17 https://www.themacallan.com/en/distilling-scotland
18 https://www.themacallan.com/en/the-macallan-chevron-highball
19 https://www.themacallan.com/en/chevron-tumblers
20 https://www.themacallan.com/en/the-macallan-chevron-water-jug
21 https://www.themacallan.com/en/the-macallan-hip-flask
22 https://www.themacallan.com/en/the-macallan-ice-ball-maker
23 https://www.themacallan.com/en/the-macallan-in-lalique-glass
24 https://www.themacallan.com/en/the-macallan-lalique-tumbler
25 https://www.themacallan.com/en/the-macallan-polo-shirt
26 https://www.themacallan.com/en/the-macallan-rocks-glass-tumblers
user_100359850
Related