I want to get all urls of a sitemap (.xml) using Scrapy.
I tried to use the example of official documents of Scrapy but I don't know why it didn't worked.
import scrapy
class XSpider(scrapy.spiders.SitemapSpider):
name = 'x'
allowed_domains = ['xxx.xxx']
sitemap_urls = ['https://www.xxx.xxx/sitemaps/sitemap_1.xml']
def parse(self, response):
print(response)
def parse_sitemap_url(self, response):
print(response)
Logs:
2021-09-14 23:25:55 [scrapy.core.engine] INFO: Spider opened
2021-09-14 23:25:57 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-09-14 23:25:57 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-09-14 23:26:57 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
XML File:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
<url>
<loc>https://www.xxx.xxx/xxxxxx</loc>
<priority>0.5</priority>
</url>
...
What should I do after this?