How to scrape data from a gchart using BS or selenium

Question

I'm looking to pull data from a chart that is viewable when you hover over it with your mouse but does not appear on the source code. Is there any way to extract this? I'm new to this but havent been able to find a discussion on this anywhere so far. It seems like a job for selenium or maybe something cool I've never heard of.

The code below relates to the cannabinoids section, which is the area I'm interested in, along with the terpenes section which has the same issue.

https://www.dinafem.org/en/amnesia-cbd/

user_6106791 · Answer

Those tables come from a POST request. You can mimic that with pure requests.

Here's how:

import json

import requests
from bs4 import BeautifulSoup

url = "https://www.dinafem.org/en/amnesia-cbd/?PageSpeed=noscript"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/90.0.4430.85 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest"
}

with requests.Session() as connection:
    product_sku = BeautifulSoup(
        connection.get(url).text,
        "html.parser",
    ).find("input", {"name": "product_sku"})["value"]

    chart_ids = ["product_cannabinoids", "product_terpenes"]

    payload = {
        "action": "get_gchart_data",
        "filters": f"product_sku={product_sku}",
    }

    charts = []
    for chart_id in chart_ids:
        payload.update({"chart_id": chart_id})
        charts.append(connection.post(url, headers=headers, data=payload).json())

print(json.dumps(charts[0], indent=2))

Output:

{
  "error": 0,
  "chart_id": "product_cannabinoids",
  "data": [
    [
      "Cannabinoid",
      "(%)"
    ],
    [
      "CBD",
      11.64
    ],
    [
      "THC",
      7.37
    ],
    [
      "CBC",
      0.7
    ],
    [
      "CBG",
      0.46
    ],
    [
      "THCV",
      0.04
    ],
    [
      "CBN",
      0.01
    ]
  ],
  "debug": {
    "total": 20.220000000000002
  }
}

You can even plot that to get a real-like chart that's on the website.

Add this to your imports:

import matplotlib.pyplot as plt

and then put this:

data = [i[1] for i in charts[1]["data"]]
info = [i[0] for i in charts[1]["data"]]
fig = plt.figure(figsize=(10, 7))
plt.pie(data, labels=info)
plt.show()

To get this:

user_3485434 · Answer

To do this with Selenium first you need to hover over the required chart rectangle.
For example, if you are looking for the second rectangle it is located by (//div[@data-chart_id='product_cannabinoids']//*[name()='g']//*[name()='g']//*[name()='rect' and contains(@width,'108')])[2]
To hover over element you can use following method:

public void hoverOverElement(By.xpath(xpath)){
        action = new Actions(driver);
        webElement = driver.findElement(element);
        action.moveToElement(webElement).build().perform();
    }

where xpath is as presented above.
Now you can locate the appeared tooltip element located by //*[name()='g' and @class='google-visualization-tooltip'] and extract it's value.

How to scrape data from a gchart using BS or selenium

2 Answers