Trouble in connecting scrapy with Firebase

Viewed 53

I'm new to scrapy and want to connect it to firestore so that I may save my results there.

Here is my settings.py (just mentioning related code)

ITEM_PIPELINES = {
         'flights_i_spider.pipelines.FirebasePipeline': 300,
 }

Is there anything else related to firebase that needs to be included to the settings.py file? because ITEM_PIPELINES was just added to settings.py.

here is my pipelines.py:

import firebase_admin
from firebase_admin import credentials, firestore
from google.cloud import firestore


class FirebasePipeline:
    def __init__(self):
        cred = credentials.Certificate('./service_key.json')
        firebase_admin.initialize_app(cred,
                                      {
                                          'databaseURL': 'https://abcdefghijklmno.firebaseio.com'
                                      }
                                      )

    def process_item(self, item, spider):
        db = firestore.Client()
        doc_ref = db.collection('output_data')
        doc_ref.add(item.to_dict())
        # doc_ref.insert(dict(item))
        return item

Please don't mention this https://github.com/skhaz/scrapy-firebase because this did not work for me.

0 Answers
Related