Is there any example about how to use dataclass and scrapy items?

user_8924894 0
Viewed 748

I was checking the documentation in Scrapy about it: https://docs.scrapy.org/en/latest/topics/items.html#dataclass-objects

from dataclasses import dataclass

@dataclass
class CustomItem:
    one_field: str
    another_field: int

Unfortunately, they don't provide any example about how to use it in spiders. I am defining my scrapy item using this syntax and I am calling in this way in my code:

        product = ProductItem(
            publicName=response.xpath(
                '//title/text()').re(r"(.+?)(?=( v?(CC-V)?(\d+\.?){1,3}))")[0],
            description=None,
            productType=self.get_product_type(response),
            salesPage=response.xpath(
                'normalize-space(//div[@class="sales_page"]/center/a/@href)').get(),
            tags=[],
            developer=response.xpath(
                '//div[@class="website"]/a/div/div/div[@class="head"]/text()').get(),
            developerId=response.xpath(
                '//div[@class="fox-sidebar"]/div[4]/div/ul/li/span/a[@rel="tag"]/@href').re(r"\/vendor\/(.*)\/")[0]
        )

I receive this error: TypeError: 'ProductItem' object is not iterable

PS: I expect to receive an error if for example define a class value as a string when an integer was expected using scrapy.

scrapy-item python-dataclasses scrapy python
1 Answers

After some trial and error I achieved the expected behavior with the help of the python library dataclass-type-validator:

spider code:

        product = ProductItem(
            publicName=response.xpath(
                '//title/text()').re(r"(.+?)(?=( v?(CC-V)?(\d+\.?){1,3}))")[0],
            productType=self.get_product_type(response.xpath(
                '//div[@class="pagination-wrapper"]/ul/li/a/text()').get()),
            salesPage=response.xpath(
                'normalize-space(//div[@class="sales_page"]/center/a/@href)').get(),
            developer=response.xpath(
                '//div[@class="website"]/a/div/div/div[@class="head"]/text()').get(),
            developerId=response.xpath(
                '//div[@class="fox-sidebar"]/div[4]/div/ul/li/span/a[@rel="tag"]/@href').re(r"\/vendor\/(.*)\/")[0],
        )
        yield { "product": product.__dict__ }

It is very important that you convert the ProductItem instance to a dict, or you will not be able to use this Item in your pipeline with an adapter.

items.py code:

from typing import List
from dataclasses import dataclass, field
from dataclass_type_validator import dataclass_validate

@dataclass_validate()
@dataclass()
class ProductItem():
    publicName: str
    productType: str
    salesPage: str
    developer: str
    developerId: str
    tags: List[str] = field(default_factory=list)
    description: str = ""
user_89248940
Related