Adding attributes to exported XML in Scrapy

user_5894294 0
Viewed 435

I am able to scrape data off a website but I need to export it in XML.

For this purpose I defined an serializer like so:

class Person(scrapy.Item):
    Name = scrapy.Field(serializer=serialize_name)
    Location = scrapy.Field()

And an XMLExportPipeline like so:

class XmlExportPipeline(object):

def __init__(self):
    self.files = {}

@classmethod
def from_crawler(cls, crawler):
    pipeline = cls()
    crawler.signals.connect(pipeline.spider_opened, signals.spider_opened)
    crawler.signals.connect(pipeline.spider_closed, signals.spider_closed)
    return pipeline


def spider_opened(self, spider):
    file = open('%s_people.xml' % spider.name, 'w+b')
    self.files[spider] = file
    self.exporter = XmlItemExporter(file, item_element='Person', root_element='People')
    self.exporter.start_exporting()


def spider_closed(self, spider):
    ...


def process_item(self, person, spider):
    self.exporter.export_item(person)
    return person

This works and gives me an XML file like so:

<?xml version="1.0" encoding="utf-8"?>
<People><Person><Name>Bob</Name><Location>NYC</Location></Person></People>

How would I add attributes to tags? For instance, if I want 

<Person Age="25" Likes="Programming">

how would I go about doing that?

Also quick follow-ups, why is the output XML not formatted like it is supposed to be? Can I convert the values in tags to CDATA (using a custom serializer to do that currently)?

scrapy xml python
2 Answers

The default implementation of XmlItemExporter does not allow this because of this line (scrapy/exporters.py:173):

self.xg.startElement(name, {})

This second argument is supposed to contain the attributes of each new element. So a workaround is to implement your own subclass of XmlItemExporter, adding this argument.

from scrapy.exporters import six, is_listlike, XmlItemExporter

class AttrXmlItemExporter(XmlItemExporter):

    def _export_xml_field(self, name, serialized_value, depth):
        # Custom code:
        attrs = {}
        if isinstance(serialized_value, dict):
            serialized_value = serialized_value.copy()
            attr_keys = [k for k in serialized_value.keys() if k.startswith('_')]
            attrs = {k[1:]: serialized_value.pop(k) for k in attr_keys}

        # Default implementation (except for startElement call)
        self._beautify_indent(depth=depth)
        self.xg.startElement(name, attrs)
        if hasattr(serialized_value, 'items'):
            self._beautify_newline()
            for subname, value in serialized_value.items():
                self._export_xml_field(subname, value, depth=depth + 1)
            self._beautify_indent(depth=depth)
        elif is_listlike(serialized_value):
            self._beautify_newline()
            for value in serialized_value:
                self._export_xml_field('value', value, depth=depth + 1)
            self._beautify_indent(depth=depth)
        elif isinstance(serialized_value, six.text_type):
            self._xg_characters(serialized_value)
        else:
            self._xg_characters(str(serialized_value))
        self.xg.endElement(name)
        self._beautify_newline()

In this example, any item value that is a dict (that is, a sub-dict) whose key starts with an underscore (_) will be rendered as an attribute.

For example, the item:

yield {
    'name': 'Sample',
    'rating': {
        '_rating': '4.5',
        '_max': '5',
    },
}

Will be rendered to XML as:

<item>
  <nam>Sample</name>
  <rating rating="4.5" max="5">
  </rating>
</item>

Haven't figured out a way to make it a self-closing element, though. Please note that all value marked as an attribute must be strings.

user_3179710

Regarding one of your follow up questions: when you use a custom XMLItemExporter, I believe the FEED_EXPORT_INDENT setting is overridden with the kwarg from when you init the XMLItemExporter. If you pass nothing, indent is None. You need to pass in kwarg for indent=<number> when you create it. You can do that in your original code along side kwargs for root and item elements, or if your using the code from the accepted answer, just override __init__ with one call to super with those kwargs.

Regarding the point about self-closing tags: it looks like the solution is another kwarg, short_empty_elements, this time to the XMLGenerator with signature xml.sax.saxutils.XMLGenerator(out=None, encoding='iso-8859-1', short_empty_elements=False). I haven't used this functionality but it looks straightforward, docs here.

user_30239580
Related