TypeError: 'NoneType' object is not iterable when using Scrapy to crawl API

user_13284700 0
Viewed 277

I am trying to start on this page (https://api.pointsbet.com/api/v2/competitions/7176/events/featured), grab the key and append it to an event api (https://api.pointsbet.com/api/v2/events/) so I can grab some data from the event api and then loop through.

I am getting the error TypeError: 'NoneType' object is not iterable and I have no idea why.

Thanks in advance!

import scrapy
import json


class PbSpider(scrapy.Spider):
    name = 'pb'
    allowed_domains = ['api.pointsbet.com']
    start_urls = ['https://api.pointsbet.com/api/v2/competitions/7176/events/featured']

    def parse(self, response):
        resp = json.loads(response.body)
        events = resp.get('events')

        for event in events:
            eventKey = event.get('key')
            gameapi = f'https://api.pointsbet.com/api/v2/events/{eventKey}'

            if gameapi:

                yield scrapy.Request(url=gameapi, callback=self.parse)

                res = json.loads(response.body)
                fixedOddsMarkets = res.get('fixedOddsMarkets')

                for fixedOddsMarket in fixedOddsMarkets:
                    market = fixedOddsMarket.get('eventName')
                    outcomes = fixedOddsMarket.get('outcomes')

                    for outcome in outcomes:
                        name = outcome.get('name')
                        price = outcome.get('price')

                        print(market, name, price)
scrapy python
1 Answers

Could you give a bit more information? I see in your code at least 3 objects for iterations (events, fixedOddsMarkets, outcomes). Which one returns an error?

Error 'NoneType' object is not iterable means that one of your .get() methods could not found an element. To avoid this error you can add the second argument into your .get() methods (it would be like resp.get('events', [])) or resp.get('events', {}). It depends on the stucture of your json (list or dict)).

But if you have some problems with connection to the api (or incorrectly assigned some json object as well) - it could not resolve your problem.

First of all, check if the response status code is acceptable (200, NOT 400, 403, etc.). You can do it with print(response.status) in your def parse(self, response) method).

If everything is OK, check json data that api returns to you directly in the code (you can do it with print(resp)). If on this step you have appropriate json file from the api (and data is OK for you), check your code and json-file structure. fixedOddsMarkets or outcomes have to be incorrectly assigned.

As I understood, it tries to scrape 'events' from your gameapi link, right? The problem is that you use 'self.parse' as callback argument.

Add another method, it should solve your problem:

def parse(self, response):
    resp = json.loads(response.body)
    events = resp.get('events')

    for event in events:
        eventKey = event.get('key')
        gameapi = f'https://api.pointsbet.com/api/v2/events/{eventKey}'

        if gameapi:

            yield scrapy.Request(url=gameapi, callback=self.parse_event)
def parse_event(self, response):
    res = json.loads(response.body)
    fixedOddsMarkets = res.get('fixedOddsMarkets')

    for fixedOddsMarket in fixedOddsMarkets:
        market = fixedOddsMarket.get('eventName')
        outcomes = fixedOddsMarket.get('outcomes')

        for outcome in outcomes:
            name = outcome.get('name')
            price = outcome.get('price')

            print(market, name, price)

Also, it is not a good variant to use if gameapi statement as it is always True. I think it is better to replace if gameapi: with if eventKey:

user_133126380
Related