Click a Button in Scrapy

user_651894 0
Viewed 72546

I'm using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it's not exactly what I need.

How can I simply click a button, which then shows the information I need?

Do I have to use an external library like mechanize or lxml?

scrapy web-scraping web-crawler python
4 Answers

Although it's an old thread I've found quite useful to use Helium (built on top of Selenium) for this purpose and far more easier/simpler than using Selenium. It will be something like the following:

from helium import *

start_firefox('your_url')
s = S('path_to_your_button')
click(s)
...
user_98363330
Related