How to scrape JSON web pages

user_11710884 0
Viewed 701

Hey so I have some experience scraping html but never json and so I need to scrape the following web page using scrapy, http://www.starcitygames.com/buylist/search?search-type=category&id=5061, and I found a tutorial online that uses scrapy along with jmspath to scrape json data from the web. And I got the tutorial to work but I am trying to alter it to work with my website to no luck. No errors but it does not return any data. Any help would be greatly appreciated!

items.py

import scrapy


class NameItem(scrapy.Item):
    """User item definition for jsonplaceholder /LoginSpider endpoint."""
    name = scrapy.Field()
    condition = scrapy.Field()
    price = scrapy.Field()
    rarity = scrapy.Field()

LoginSpider.py

import scrapy
import json
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import NameItem
from scrapy.loader import ItemLoader
from scrapy.loader.processors import Join, MapCompose, SelectJmes


class UserSpider(scrapy.Spider):
    """Spider to scrape `http://www.starcitygames.com/buylist/search?search-type=category&id=5061`."""
    name = 'LoginSpider'
    allowed_domains = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    start_urls = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    # dictionary to map UserItem fields to Jmes query paths
    jmes_paths = {
            'name': 'name',
            'condition': 'condition',
            'price': 'price',
            'rarity': 'rarity',
            }

    def parse(self, response):
        jsonresponse = json.loads(response.body_as_unicode())
        for user in jsonresponse:
            loader = ItemLoader(item=NameItem())  # create an ItemLoader to populate a NameItem
            loader.default_input_processor = MapCompose(str)  # apply str conversion on each value
            loader.default_output_processor = Join(' ')
            for (field, path) in self.jmes_paths.items():
                loader.add_value(field, SelectJmes(path)(user))
            yield loader.load_item()
scrapy python json
1 Answers

The response of this url http://www.starcitygames.com/buylist/search?search-type=category&id=5061has 3 levels:

  1. 'Ok'
  2. 'search'
  3. 'results' ## this contain the data

And results key has multiple values what you should iterate. Inside the values are the data. Try this code, I hope you can help.

This is the module items.py

class SoResponseItem(scrapy.Item):
        name = scrapy.Field()
        condition = scrapy.Field()
        price = scrapy.Field()
        rarity = scrapy.Field()

This is the spider

import scrapy
import json
from SO_response.items import SoResponseItem

class LoginspiderSpider(scrapy.Spider):
    name = 'LoginSpider'
    allowed_domains = ['www.starcitygames.com']
    url = 'http://www.starcitygames.com/'

    def start_requests(self):
        yield scrapy.Request(url=self.url, callback=self.parse)

    def parse(self, response):
        url = response.urljoin('buylist/search?search-type=category&id=5061')
        yield scrapy.Request(url=url, callback=self.parse_data)

    def parse_data(self, response):
        jsonreponse = json.loads(response.body)
        for result in jsonreponse['results']:
            for index in range(len(result)):
                items = SoResponseItem()
                items['name'] = result[index]['name']
                items['condition'] = result[index]['condition']
                items['price'] = result[index]['price']
                items['rarity'] = result[index]['rarity']
                yield items

Try in your shell: scrapy crawl -o jmes.json

user_111016090
Related