Can't grab tabular content from a webpage using requests

user_10568531 0
Viewed 88

I'm trying to scrape tabular content from this webpage. To locate the content, it is necessary to click on the 12 tab under this title How to navigate the interactive report. Upon clicking on that tab, the tabular content shows up at the bottom of that page under Moves To Austin-Round Rock-Georgetown, TX MSA.

When I observe network activity in chrome dev tools while populating the data manually, I could notice that a post http requests along with appropriate parameter is sent to this url https://public.tableau.com/vizql/w/CBREMigrationAnalysisv1extract/v/CBREMigrationAnalysis/sessions/F3E2227B603E4F5AB3156667A673CF9E-0:0/commands/tabdoc/set-active-story-point in which the portion between /sessions/ and /commands/ is dynamic.

However, I have been able to fetch that dynamic portion from this url on the fly before sending the post requests. Now, when I try with the following script, I get 500 status code.

I've tried with:

import requests
from bs4 import BeautifulSoup
from pprint import pprint

base = 'https://public.tableau.com/views/CBREMigrationAnalysisv1extract/CBREMigrationAnalysis?:showVizHome=no&:embed=true&parentUrl=https%3A%2F%2Fwww.cbre.us%2Fresearch-and-reports%2FCOVID-19-Impact-on-Resident-Migration-Patterns'
link = 'https://public.tableau.com/vizql/w/CBREMigrationAnalysisv1extract/v/CBREMigrationAnalysis/sessions/{}/commands/tabdoc/set-active-story-point'

payload = {
    'storyboard': 'CBRE Migration Analysis',
    'storyPointId': '14',
    'shouldAutoCapture': 'false',
    'shouldAutoRevert': 'true'
}

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36',
    'x-requested-with': 'XMLHttpRequest',
    'x-newrelic-id': 'XA4CV19WGwIBV1RVBQQBUA==',
    'x-tsi-active-tab': 'CBRE%20Migration%20Analysis',
    'x-tsi-supports-accepted': 'true',
    'referer': base,
}
with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    r = s.get(base)
    post_link = link.format(r.headers['X-Session-Id'])
    s.headers.update(headers)
    res = s.post(post_link,data=payload)
    print(res.status_code)
    pprint(res.json()['vqlCmdResponse']['layoutStatus']['applicationPresModel'])

How can I access tabular content from that page using requests?

python-requests web-scraping python-3.x python
1 Answers

I've just implemented the storypoints feature in this Tableau Scraper library. Checkout the storypoint section

The following code will show all the storypoints, and go to storypoint with id 14 (equivalent to the storypoint with caption 12 in the UI). Then it gets the worksheet with name P2P Table into a pandas dataframe:

from tableauscraper import TableauScraper as TS

url = 'https://public.tableau.com/views/CBREMigrationAnalysisv1extract/CBREMigrationAnalysis'
ts = TS()
ts.loads(url)
wb = ts.getWorkbook()

print(wb.getStoryPoints())
print("go to specific storypoint")
sp = wb.goToStoryPoint(storyPointId=14)

print(sp.getWorksheetNames())
print(sp.getWorksheet("P2P Table").data)

Try this on repl.it

user_26143640
Related