I'm trying to crawl a directory website (something like yellow pages) that requires a log-in to see the information in each specific listing but I ran into a problem and it seems like the spider loses authentication every other listing crawled.
There are also cases where the directory listing redirects to another domain with similar structure so it can still be crawled the same way. The difference only is that it is redirected then checks for authentication on the original website and then passes the auth to the new website (not entirely sure about this I'm not really versed in web auth processes).
I've tried putting CONCURRENT_REQUESTS = 1 in my settings.py and it doesn't make much of a difference except that sometimes it successfully passes the authentication 2 listings in a row.
It seems like including or not including the __RequestVerificationToken does not make a difference.
I believe I have followed Scrapy's docs on logging in before scraping but if there's anything I might have missed please help me single it out too.
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = START_URL
def parse(self, response):
csrf_token = response.xpath('//*[@name="__RequestVerificationToken"]/@value').extract_first()
yield scrapy.FormRequest.from_response(response,
formid='login-form',
clickdata={'type': 'submit'},
formdata={
'__RequestVerificationToken': csrf_token,
'username':'user',
'password': 'pass'},
callback=self.parse_after_login)
def parse_after_login(self, response):
if b"Member Section" in response.body:
self.log("Successfully logged in. Let's start crawling!")
# Now the crawling can begin..
for url in PAGES:
yield scrapy.Request(url, callback=self.parse_directory_page)
else:
self.log("Bad times :(")
# Something went wrong, we couldn't log in, so nothing happens.
def parse_directory_page(self, response):
print("DETECTED: " + str(response.css('#navright > div.link_login.entry > a::text').extract()) + " BUTTON IN DIRECTORY LISTING PAGE")
...URL Manipulation code
for listing in response.css(LISTING_SELECTOR):
yield scrapy.Request(listing, self_parse_listing)
def parse_listing(self, response):
print("DETECTED: " + str(response.css('#navright > div.link_login.entry > a::text').extract()) + " BUTTON IN COMPANY PAGE")
...scrape and data manipulation
Console Log with COOKIES_DEBUG=True
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com/MemberSection> (referer: https://www.example.com/Account/Login)
[myspider] DEBUG: Successfully logged in. Lets start crawling!
[scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://www.example.com/Directory?siteID=24&au=m&pageIndex=1&pageSize=100&searchby=CountryCode&country=NZ&city=&keyword=&orderby=CountryCity&networkIds=1&networkIds=2&networkIds=3&networkIds=4&networkIds=61&networkIds=98&networkIds=108&networkIds=5&networkIds=22&networkIds=13&networkIds=18&networkIds=15&networkIds=16&networkIds=38&networkIds=103&layout=v1&submitted=search>
Cookie: ASP.NET_SessionId=lgvbj33sbnlbsyp4hpe5jhxf;
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com/Directory?siteID=24&au=m&pageIndex=1&pageSize=100&searchby=CountryCode&country=NZ&city=&keyword=&orderby=CountryCity&networkIds=1&networkIds=2&networkIds=3&networkIds=4&networkIds=61&networkIds=98&networkIds=108&networkIds=5&networkIds=22&networkIds=13&networkIds=18&networkIds=15&networkIds=16&networkIds=38&networkIds=103&layout=v1&submitted=search> (referer: https://www.example.com/MemberSection)
DETECTED: ['SIGN OUT'] BUTTON IN DIRECTORY LISTING PAGE
[scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 https://example.com/directory/members/123640>
Set-Cookie: ASP.NET_SessionId=iyeclhcbqwxbfycplifedyxu; path=/; HttpOnly; SameSite=Lax
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.com/directory/members/123640> (referer: https://www.example.com/Directory?siteID=24&au=m&pageIndex=1&pageSize=100&searchby=CountryCode&country=NZ&city=&keyword=&orderby=CountryCity&networkIds=1&networkIds=2&networkIds=3&networkIds=4&networkIds=61&networkIds=98&networkIds=108&networkIds=5&networkIds=22&networkIds=13&networkIds=18&networkIds=15&networkIds=16&networkIds=38&networkIds=103&layout=v1&submitted=search)
[scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://example.com/directory/members/119259>
Cookie: ASP.NET_SessionId=iyeclhcbqwxbfycplifedyxu
DETECTED: ['SIGN IN'] BUTTON IN COMPANY PAGE
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://webservice.example.com/wcasso/SsoV1/CheckLoggedIn?otherParams> from <GET https://example.com/directory/members/119259>
[scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://webservice.example.com/wcasso/SsoV1/CheckLoggedIn?otherParams>
Cookie: ASP.NET_SessionId=c2e3avij54xfpn4a5ofl4xdt;sso_signin=user_id=user_id_code;ASP.NET_SessionId=iyeclhcbqwxbfycplifedyxu
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.example.com/Account/SsoLoginResult?user_id=000001&login_token=b612f5378f494713&p_action=&p_cid=0&returnurl=https%3a%2f%2fexample.com%2fdirectory%2fmembers%2f119259> from <GET https://webservice.example.com/wcasso/SsoV1/CheckLoggedIn?random=202104200004375582&otherParams>
[scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://www.example.com/Account/SsoLoginResult?Params>
Cookie: ASP.NET_SessionId=lgvbj33sbnlbsyp4hpe5jhxf; __RequestVerificationToken=TokenHere; ASP.NET_SessionId=iyeclhcbqwxbfycplifedyxu
*I removed/changed some information from the debug since stackoverflow seems to think it's spam.