web scraping vaadin python

user_10923560 0
Viewed 686

I'm trying to scrape a site created using vaadin using python. This is the code I use:

requests.get('http://rnb.osim.ro/?pn=').text

but this is the result which contains no useful information:

    <noscript>
    You have to enable javascript in your browser to use an application built with Vaadin.
   </noscript>
  </div>
  <script type="text/javascript" src="./VAADIN/vaadinBootstrap.js"></script>
  <script type="text/javascript">//<!

Do you know how I can get the data I need from a vaadin site?

python-requests vaadin web-scraping python
2 Answers

This is happening because requests can't really execute the JavaScript inside the website. As you might think, requests is just a request and not a browser that can handle JS and work with frameworks for the front end (i.e. Angular, React, Ajax). To scrape this modern and robust websites I personally recommend to use scrapy library. It's designed specially for scraping and can handle with the JavaScript a little bit. And even if it couldn't, you can still use the selenium web driver to fully emulate a browser.

user_71160390

If you are already familiar with requests, you may also find requests-html useful. If your just want to get the rendered html and not interact with the page, like clicking buttons, page-down etc. then you can use this option.

Your question is ideal for this demonstration. The following code fully renders the html you want.

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('http://rnb.osim.ro/?pn=')
r.html.render(sleep = 5)
print(r.html.html)
user_108255560
Related