Issue scraping website with bs4 (beautiful soup) python 2.7

user_11335084 0
Viewed 73

What I am attempting to accomplish is a simple python web scraping script for google trends and running into an issue when grabbing the class

from bs4 import BeautifulSoup
import requests


results = requests.get("https://trends.google.com/trends/trendingsearches/daily?geo=US")
soup = BeautifulSoup(results.text, 'lxml')
keyword_list = soup.find_all('.details-top')
for keyword in keyword_list:
    print(keyword)

When printing tag I receive and empty class however when I print soup I receive the entire HTML document. My goal is to print out the text of each "Keyword" that was searched for the page https://trends.google.com/trends/trendingsearches/daily?geo=AU

this has a list of results:

1. covid-19
2.Woolworths jobs

If you use google developer options select inspect and hover over the title you will see div.details-top.

how would I just print the text of the title of each

python-2.7 web-scraping beautifulsoup
1 Answers

I can see that data being dynamically retrieved from an API call in the dev tools network tab. You can issue an xhr to that url then use regex on the response text to parse out the query titles.

import requests, re
from bs4 import BeautifulSoup as bs

r = requests.get('https://trends.google.com/trends/api/dailytrends?hl=en-GB&tz=0&geo=AU&ns=15').text
p = re.compile(r'"query":"(.*?)"')
titles = p.findall(r)
print(titles) # 2.7 use print titles
user_62412350
Related