Parsing an 'Load More' response with HTML content

Question

I'm trying to scrape each content in Istanbul Governorate's announcement section located at the link below, which loads content with a 'Load More' at the bottom of the page. From dev tools / Network, I checked properties of the POST request sent and updated the header accordingly. The response apparently is not json but an html code.

I would like to yield the parsed html responses but when I crawl it, it just doesn't return anything and stuck with the first request forever. Thank you in advance.

Could you explain me what's wrong with my code? I checked tens of questions here but couldn't resolve the issue. As I understand, it just can't parse the response html but I couldn't figure out why.

ps: I have been enthusiastically into Python and scraping for 20 days. Forgive my ignorance.

import scrapy

class DuyurularSpider(scrapy.Spider):
    name = 'duyurular'
    allowed_domains = ['istanbul.gov.tr']
    start_urls = ['http://istanbul.gov.tr/duyurular']

    headerz = {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Accept-Language": "en-US,en;q=0.9",
        "Connection" : "keep-alive",
        "Content-Length": "112",
        "Content-Type": "application/json",
        "Cookie" : "_ga=GA1.3.285027250.1638576047; _gid=GA1.3.363882495.1639180128; ASP.NET_SessionId=ijw1mmc5xrpiw2iz32hmqb3a; NSC_ESNS=3e8876df-bcc4-11b4-9678-e2abf1d948a7_2815152435_0584317866_00000000013933875891; _gat_gtag_UA_136413027_31=1",
        "Host": "istanbul.gov.tr",
        "Origin": "http://istanbul.gov.tr",
        "Referer": "http://istanbul.gov.tr/duyurular",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
        "X-Requested-With": "XMLHttpRequest",
                }

    def parse(self, response):

        url = 'http://istanbul.gov.tr/ISAYWebPart/Announcement/AnnouncementDahaFazlaYukle'
        load_more = scrapy.Request(url, callback = self.parse_api, method = "POST", headers = self.headerz)    

        yield load_more    

    def parse_api(self, response):
        raw_data = response.body

        
        data = raw_data.xpath('//div[@class="ministry-announcements"]')

        for bilgi in data:

            gun =  bilgi.xpath('//div[@class = "day"]/text()').extract_first()  #day
            ay = bilgi.xpath('//div[@class = "month"]/text()').extract_first() #month

            metin = bilgi.xpath('//a[@class ="announce-text"]/text()').extract_first() #text

            yield {'Ay:' : ay,
                   'Gün' : gun,
                   'Metin': metin,}

Result I encounter:

user_16429780 · Answer

Remove Content-Length, also never include it in the headers. Also you should remove the cookie and let scrapy handle it.
Look at the request body and recreate it for every page:
You need to know when to stop, in this case it's an empty page.
in the bilgi.xpath part you're getting the same line over and over because you forgot a dot at the beginning.

The complete working code:

import scrapy
import json


class DuyurularSpider(scrapy.Spider):
    name = 'duyurular'
    allowed_domains = ['istanbul.gov.tr']
    start_urls = ['http://istanbul.gov.tr/duyurular']
    page = 1
    
    headers = {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Accept-Language": "en-US,en;q=0.9",
        "Connection": "keep-alive",
        "Content-Type": "application/json",
        "Host": "istanbul.gov.tr",
        "Origin": "http://istanbul.gov.tr",
        "Referer": "http://istanbul.gov.tr/duyurular",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
        "X-Requested-With": "XMLHttpRequest",
    }

    def parse(self, response):
        url = 'http://istanbul.gov.tr/ISAYWebPart/Announcement/AnnouncementDahaFazlaYukle'
        body = {
            "ContentCount": "8",
            "ContentTypeId": "D6mHJdtwBYsvtS2xCvXiww==",
            "GosterimSekli": "1",
            "OrderByAsc": "true",
            "page": f"{str(self.page)}"
        }

        if response.body.strip():    # check if we get an empty page
            load_more = scrapy.Request(url, method="POST", headers=self.headers, body=json.dumps(body))
            yield load_more
            self.page += 1

            data = response.xpath('//div[@class="ministry-announcements"]')
            for bilgi in data:
                gun = bilgi.xpath('.//div[@class = "day"]/text()').extract_first()  #day
                ay = bilgi.xpath('.//div[@class = "month"]/text()').extract_first() #month

                metin = bilgi.xpath('.//a[@class ="announce-text"]/text()').extract_first() #text

                yield {
                    'Ay:': ay,
                    'Gün': gun,
                    'Metin': metin.strip(),
                }

Parsing an 'Load More' response with HTML content

1 Answers