I am trying to create get my Scrapy program to login, but it isn't working.
import scrapy
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
class StudentSpider(scrapy.Spider):
name = "students"
def start_requests(self):
login = r"https://iin.stage.edlumina.com"
return [scrapy.Request(url=login, callback=self.login)]
def login(self, response):
token = response.css("form input[name='authenticity_token']::attr(value)").extract_first()
print("\nToken found:", token)
formdata = {
"user[email]": "email",
"user[password]": "password",
}
return FormRequest.from_response(response, formdata=formdata, callback = self.scrape_dash)
def scrape_dash(self, response):
print("Logged in!")
print("Heres some user data:", response.body)
url = r"https://iin.stage.edlumina.com/program_registrations"
yield scrapy.Request(url=url, callback=self.parse_dash)
def parse_dash(self, response):
open_in_browser(response)
These are the logs that are returned when I try to run the program:
2022-06-16 11:34:15 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: edlumina)
2022-06-16 11:34:15 [scrapy.utils.log] INFO: Versions: lxml 4.9.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.10.5 (tags/v3.10.5:f377153, Jun 6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)], pyOpenSSL 22.0.0
(OpenSSL 3.0.3 3 May 2022), cryptography 37.0.2, Platform Windows-10-10.0.19044-SP0
2022-06-16 11:34:15 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'edlumina',
'NEWSPIDER_MODULE': 'edlumina.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['edlumina.spiders']}
2022-06-16 11:34:15 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-06-16 11:34:15 [scrapy.extensions.telnet] INFO: Telnet Password: <omitted>
2022-06-16 11:34:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-06-16 11:34:16 [scrapy.core.engine] INFO: Spider opened
2022-06-16 11:34:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-06-16 11:34:16 [scrapy.extensions.telnet] INFO: Telnet console listening on <omitted>
2022-06-16 11:34:16 [filelock] DEBUG: Attempting to acquire lock 1427280154336 on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Lock 1427280154336 acquired on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Attempting to release lock 1427280154336 on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Lock 1427280154336 released on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://iin.stage.edlumina.com/robots.txt> (referer: None)
2022-06-16 11:34:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://iin.stage.edlumina.com> (referer: None)
Token found: <omitted>
2022-06-16 11:34:17 [scrapy.core.engine] DEBUG: Crawled (201) <POST https://iin.stage.edlumina.com/login.json> (referer: https://iin.stage.edlumina.com)
Logged in!
Heres some user data: b'{"user":{"first_name":<omitted>,"last_name":<omitted>,"email":<omitted>,"authentication_token":<omitted>}}'
2022-06-16 11:34:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://iin.stage.edlumina.com/> from <GET https://iin.stage.edlumina.com/program_registrations>
2022-06-16 11:34:17 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://iin.stage.edlumina.com/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2022-06-16 11:34:17 [scrapy.core.engine] INFO: Closing spider (finished)
2022-06-16 11:34:17 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1457,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 3,
'downloader/request_method_count/POST': 1,
'downloader/response_bytes': 8795,
'downloader/response_count': 4,
'downloader/response_status_count/200': 2,
'downloader/response_status_count/201': 1,
'downloader/response_status_count/302': 1,
'dupefilter/filtered': 1,
'elapsed_time_seconds': 1.145782,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 6, 16, 15, 34, 17, 435784),
'httpcompression/response_bytes': 15168,
'httpcompression/response_count': 2,
'log_count/DEBUG': 10,
'log_count/INFO': 10,
'request_depth_max': 2,
'response_received_count': 3,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2022, 6, 16, 15, 34, 16, 290002)}
2022-06-16 11:34:17 [scrapy.core.engine] INFO: Spider closed (finished)
My observations: When I send the FormRequest, the request body returns the information associated with the account as well as an authentication token, so I assume it successfully logged in. However, upon trying to go further into the site, it automatically redirects back to the home page, an action that is similar to if you were to physically delete the session cookie, making me think it might have something to do with not storing the cookie (or authentication token? ) properly. Could you take a look at the code and see if I should be doing something differently? Is Scrapy supposed to handle the session cookies or am I supposed to?