Is there a way to render a HTML page without using a browser and then scrape it's content?

Question

I need to extract some text from a webpage but the webpage is dynamically built(plugin). i.e I need to include a javascript SDK

and then place the code where I want the plugin to appear on my page

so in total, I have something like

Rendering this page on a browser should automatically load in some data which I now want to scrape. Is there a way to render this HTML in python? I've tried using

from requests_html import HTML

doc = # the content above
html = HTML(html=doc)
page = html.render(keep_page=True, sleep=120)

but the page is always None

Ideally, I would like something like

html_code = #here
loaded_html_code = a_package.render(html_code) # This should render my HTML which in turn causes an Iframe to be loaded.

user_188331 · Answer

You can use Beautiful Soup and Selenium Web Driver to achieve your goal. Here is an example code:

import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup

URL = "https://example.com/"
driver = webdriver.Firefox()
driver.get(URL)

time.sleep(15) # in seconds. 15 seconds should be enough to load the contents from API, JS, AJAX, etc.
html = driver.page_source
soup  = BeautifulSoup(html)

# find elements by ID
results = soup.find(id="target_id")

Is there a way to render a HTML page without using a browser and then scrape it's content?

1 Answers