Scraping the "Compare Vintages" from Vivino

user_7575172 0
Viewed 243

I'm trying to scrape data from Vivino, and so far I managed to use the API and read from the json file using this post: https://stackoverflow.com/a/62224619/7575172

r = requests.get(
    "https://www.vivino.com/api/explore/explore",
    params = {
        "country_code": "DK",
        "country_codes[]":"fr",
        "currency_code":"DKK",
        "grape_filter":"varietal",
        "min_rating":"1",
        "order_by":"price",
        "order":"asc",
        "page": 1,
        "price_range_max":"500",
        "price_range_min":"0",
        "wine_type_ids[]":"1",
        
    },
    headers= {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0"
    }
)
results = [
    (
        t["vintage"]["wine"]["winery"]["name"], 
        f'{t["vintage"]["wine"]["name"]}',
        t['vintage']['year'],
        t['vintage']['wine']['region']['country']['name'],
        # t['vintage']['wine']['region']['name_en'],
        t['vintage']['wine']['style']['region']['name'],
        t["vintage"]["statistics"]["ratings_average"],
        t["vintage"]["statistics"]["ratings_count"],
        t['price']['amount']
    )
    for t in r.json()["explore_vintage"]["matches"]
]
dataframe = pd.DataFrame(results,columns=
                         ['Winery',
                          'Wine',
                          'Year',
                          'Country',
                          'Region',
                          'Rating',
                          'num_review',
                          'Price'])

# print(dataframe)
print(dataframe[['Winery','Year','Region','Rating','num_review','Price']])

However, I cannot find the data that describes the other available vintages of the same wine in any of the json files. E.g. I'm looking at 2019, but data is also available for 2015-2020.

I have used the network monitor in Firefox to inspect other json files that are sent when you open the page below. But as far as I can see, the information about total available vintages is not present?

An example of the section I want to scrape can be seen here and in the image: https://www.vivino.com/DK/en/pierre-amadieu-gigondas-romane-machotte-rouge/w/73846?ref=nav-search#vintageListSection

enter image description here

web-scraping python
2 Answers

The data is located in the javascript under window.__PRELOADED_STATE__.winePageInformation object like this:

<script>
  window.__PRELOADED_STATE__ = ....
  window.__PRELOADED_STATE__.winePageInformation = { very long JSON here }
</script>

You can use a regex to extract it, and the result seems to be valid JSON:

import requests
import re
import json

url = "https://www.vivino.com/DK/en/pierre-amadieu-gigondas-romane-machotte-rouge/w/73846"
r = requests.get(url,
headers= {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0"
})
# this gets the javascript object
res = re.search(r"^.*window\.__PRELOADED_STATE__\.winePageInformation\s*=\s*(.*});", r.text, re.MULTILINE)
print( r.text)
data = json.loads(res.group(1))

print("recommended vintages")
print(data["recommended_vintages"])

print("all vintages")
print(data["wine"]["vintages"])
user_26143640

Bertrand has given you the best answer. Perhaps oddly, the endpoint you are hitting is not configured for allowing you to pass in a wine id and get all vintages back. The available params are:

country_code, country_codes, currency_code, discount_prices, food_ids, 
grape_ids, grape_filter, max_rating, merchant_id, merchant_type, min_rating,
min_ratings_count, order_by, order, page, per_page, price_range_max, 
price_range_min, region_ids, wine_style_ids, wine_type_ids, winery_ids, 
vintage_ids, wine_years, excluding_vintage_id, wsa_year, top_list_filter

These are detailed in the JS file https://www.vivino.com/packs/common-8f26f13b0ac53f391471.js.

You would need to determine the vintage ids and pass those in to the API instead.

user_62412350
Related