Scraping text injected by Javascript using BeautifulSoup

user_13832153 0
Viewed 662

I want to scrape the date from a webpage, and the text of the date (after script tag) is injected by JavaScript: I want to scrape it using Beautifulsoup only, not selenium

<div class="row">
    <span class="LName"><a target="_blank" href="http://google.com">[me too]</a></span>
    <script language="Javascript" type="text/javascript">formatDate('2020,5,23,09,00,00',1)</script>6/23/2020&nbsp;10:00&nbsp;Tuesday
</div>

this is the code used to try scraping the date text:

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'}
r = requests.get(u, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

I try :

soup.select('div.row > script')[0].get_text()

returns:

"formatDate('2020,5,23,09,00,00',1)"

and :

soup.select('div.row')[0].get_text()

returns:

"\n[me too] formatDate('2020,5,23,09,00,00',1)\n"

when I use Chrome to inspect the tag, I can see the date text after the script tag

when I execute:

soup.select('div.row')

it returns the tag without the date text

I want to scrape only with Beautifulsoup, not selenium

web-scraping beautifulsoup python
3 Answers

It's my understanding that BS does not render Javascript at all. I use Selenium to grab the content, and then BS to parse. If you insist on not using selenium, there are other packages that will render the page first, then you feed it into BeautifulSoup for parsing.

Try: https://github.com/makinacorpus/spynner PhantomJS

user_137762750

If you can rely on the date to always be generated with the formatDate() function and can assume the date display format, then you could extract the call with BeautifulSoup as you do and then use regular expressions to parse it out:

import re
date_call = soup.select('div.row > script')[0].get_text()
year, month, day, hour, minutes, seconds, dow = re.search(r"formatDate\('(\d+),(\d+),(\d+),(\d+),(\d+),(\d+)',(\d)\)", date_call).groups()

The variables will then give you the components you need to reconstruct the date:

dow_map = {str(i):dow for i, dow in enumerate(['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'])}
date_text = f'{month}/{day}/{year} {h}:{m} {dow_map[dow]}'
user_42375910 
import re
from bs4 import BeautifulSoup
from datetime import datetime


txt = '''<div class="row">
    <span class="LName"><a target="_blank" href="http://google.com">[me too]</a></span>
    <script language="Javascript" type="text/javascript">formatDate('2020,5,23,09,00,00',1)</script>6/23/2020&nbsp;10:00&nbsp;Tuesday
</div>'''

soup = BeautifulSoup(txt, 'html.parser')
year, month, day, hour, minutes, sec, *_ = map(int, re.findall(r'\d+', soup.select_one('div.row > script').contents[0]))
d = datetime(year=year, month=month + 1, day=day, hour=hour + 1, minute=minutes, second=sec)
print(datetime.strftime(d, '%m/%d/%Y %H:%M %A'))

Prints:

06/23/2020 10:00 Tuesday
user_100359850
Related