Unable to scrape data from website graph using python

encrypted 0
Viewed 33

I'm trying to scrape (parse) the data shown in a graph from https://www.poder360.com.br/agregador-de-pesquisas/.

I have tried requests, requests-html and beautifulsoup but I'm unable to parse the whole website. Even when I click the right button and view the page source, it won't show the table with the data, whose id is "method-table".

Code from last attempt:

from requests_html import HTMLSession

def get_data(url_path):
    from requests_html import HTMLSession
    session = HTMLSession()

    r = session.get(url_path)
    r.html.render(wait = 8, sleep = 8)

    return r.html

url_path = 'https://www.poder360.com.br/agregador-de-pesquisas'
content = get_data(url_path)
print(content.html)

Also trying the following code

import requests
import json
from bs4 import BeautifulSoup

url = 'https://www.poder360.com.br/agregador-de-pesquisas'

r = requests.get(url)

soup = BeautifulSoup(r.content, 'html.parser')

print(soup)
parsing web-scraping python-requests python
1 Answers

I think that is because you need Javascript to run to render the whole page and show the graph, which does not work with HTMLSession oder requests.

If you click "Inspect" in the Browser on the page and look at the live code instead of the page source, you can search for "circle" and find the data points of the graph.

Maybe this could help: Using python Requests with javascript pages

JHNT0
Related