Scrapy Pipelines to Seperate Folder/Files - Abstraction

Viewed 1295

I currently finalising a Scrapy project however I have quite a lengthy pipelines.py file.

I noticed that in my settings.py the pipe lines are show as follows (trimmed down):

ITEM_PIPELINES = {
     'proj.pipelines.MutatorPipeline': 200,
     'proj.pipelines.CalculatorPipeline': 300,
     'proj.pipelines.SaveToFilePipeline': 500,
}

I have tried the following ways to rectify this.

1.) I created a new file/folder and tried referencing it from pipeline in the same manner.

Folder was myPipelines/Test.py with a class name TestPipeline then referenced in pipeline settings as proj.myPipelines.Test.TestPipeline': 100,.

This threw me errors.

I then thought I could export the module and import into my current pipelines.py and it would take the reference from that. I added an empty __init__.py in my myPipelines directory and then added from myPipelines.Test import TestPipeline but scrapy still throws an error of...

Raise NameError("Module '%s' doesn't define any object named '%s'" % (module, name))
exceptions.NameError: Module 'proj.pipelines' doesn't define any object named 'TestPipeline'.

Many thanks in advance!

1 Answers
Related