How can I use the fields_to_export attribute in BaseItemExporter to order my Scrapy CSV data?

user_1147688 0
Viewed 10622

I have made a simple Scrapy spider that I use from the command line to export my data into the CSV format, but the order of the data seem random. How can I order the CSV fields in my output?

I use the following command line to get CSV data: 

scrapy crawl somwehere -o items.csv -t csv

According to this Scrapy documentation, I should be able to use the fields_to_export attribute of the BaseItemExporter class to control the order. But I am clueless how to use this as I have not found any simple example to follow.

Please Note: This question is very similar to THIS one. However, that question is over 2 years old and doesn't address the many recent changes to Scrapy and neither provides a satisfactory answer, as it requires hacking one or both of:

to address some previous issues, that seem to have already been resolved...

Many thanks in advance.

scrapy csv python
3 Answers

I found a pretty simple way to solve this issue. The above answers I would still say are more correct, but this is a quick fix. It turns out scrapy pulls the items in alphabetical order. Capitals are also important. So, an item beginning with 'A' will be pulled first, then 'B', 'C', etc, followed by 'a', 'b', 'c'. I have a project going right now where the header names are not extremely important, but I did need the UPC to be the first header for input into another program. I have the following item class:


    class ItemInfo(scrapy.Item):
            item = scrapy.Field()
            price = scrapy.Field()
            A_UPC = scrapy.Field()
            ID = scrapy.Field()
            time = scrapy.Field()

My CSV file outputs with the headers (in order): A_UPC, ID, item, price, time

user_151254030
Related