Scrapy - How to extract XmlHttpRequests url from a given website?

user_5621106 0
Viewed 1549

I'm interested to extract just the XHR's urls, not every url in a webpage: screenshot reference

Thats my code to extract every url in a page:

import scrapy
import json
from scrapy.selector import HtmlXPathSelector

from scrapy.spiders import CrawlSpider, Rule, Spider
from scrapy.linkextractors import LinkExtractor

class test(CrawlSpider):
    name = 'test'
    start_urls = ['SomeURL']
    filename = 'test.txt'

rules = (

    Rule(LinkExtractor(allow=('', )) ,callback='parse_item'),
)

def parse_item(self, response):
    # hxs = HtmlXPathSelector(response)
    with open ('test.txt', 'a') as f:
        f.write (response.url + '\n' )

Thanks,

Edited: Hi, thanks for the comments. After more research I came across this : Scraping ajax pages using python What I want is to do this answer automatically. I need to to this for a big amount of web pages and manually inserting the urls isn't an option. Is there a way to do that? Listen to the XHR requests of a site and save the urls?

scrapy web-crawler xmlhttprequest
1 Answers

There is no reliable, single way to get "the AJAX URL" of a web page. Web pages can use any number of AJAX URLs (and most of them would not be the ones you are looking for), and they can be triggered of very different ways.

Also, the URLs themselves are seldom useful, each one can returns any kind of data, and its is usually the data that you are interested in.

You should find AJAX URLs manually on a website-by-website basis.

user_9393640
Related