How to extend scrapy feed export to send crawl results in POST requests - open and store not called?

user_3298556 0
Viewed 117

I have written a spider and it works fine. I now wish to send the results of that scrape to a rest API via POST request.

I'm under the impression that extending the feed export functionality to have it make POST requests instead of writing to a file/sending it to S3/etc.

I am sure that my settings.py and configuration is correct, but for some reason the open and close functions are never called, even when I literally copy and paste and rename preexisting feed export classes from feedexport.py.

Please see code below:

settings.py

EXTENSIONS = {'project.extensions.ExportToApi.ExportScrape':400}
FEED_STORAGES = {'http': 'project.extensions.ExportToApi.ExportScrape'}
FEED_API_URL = 'http://localhost:8000'

(there's a simpleserver at localhost:8000 there printing out all the GET and POST requests it receives)

/project/extensions/ExportToApi.py

from scrapy.extensions.feedexport import BlockingFeedStorage

class ExportScrape(BlockingFeedStorage):
    def __init__(self, crawler, uri):
        self.crawler = crawler
        self.url = uri

    @classmethod
    def from_crawler(cls, crawler):
        uri_from_settings = crawler.settings['FEED_API_URL']
        return cls(crawler, uri_from_settings)

    def _store_in_thread(self, file):
        file.seek(0)
        import requests
        r = requests.post(self.url, data=file)
        file.close()

When added, print statements are run from __init__ and from_crawler, but not _store_in_thread, nor open or store if they are implemented.

python-requests scrapy python
1 Answers

POST requests to rest api - is directly the same type of http requests as regular requests scrapy send.
One of options is to.. inside parse method return Request object with empty callback to rest API endpoint instead of item object of dict (and withiout any custom feed storages)

def parse(self, response)
    ...
    #yield item
    ...
    yield Request(
        method='post',
        url=server_url,
        body=converted_item_data,
        callback=self.not_parse)

def not_parse(self, response):
    pass
user_108847910
Related