Login with authenticated session with Scrapy

Question

I am writing a web scraping project in Python using Scrapy. As a reference, my website I'm planning on scraping is https://umass.moonami.com/.

The problem is the login phase. Conventionally, when I login using a browser, it should redirect me to: https://login.microsoftonline.com/ (sending SAML request). However, in Scrapy, I can only reach to: https://webauth.umass.edu/idp/profile/SAML2/Redirect/SSO?execution=e1s1.

Can anyone help me figure why that is? Thank you very much.

user_12972198 · Answer

In most cases, login with Scrapy or other similar libraries is almost impossible. (I'm not sure about this case)

So I suggest you to use headless browsers instead, there are two famous frameworks for this purpose:

Puppeteer (my recommendation) which is a Nodejs library: https://github.com/puppeteer/puppeteer

Selenium: https://selenium-python.readthedocs.io/

They will make your job much easier, but they will consume much more resources.