Python JSON scraping - how can I handle missing values?

user_1295028 0
Viewed 534

I'm pretty new to coding, so I'm learning a lot as I go. This problem got me stumped, and even though I can find several similar questions on here, I can't find one that works or has a recognizable syntax to me.

I'm trying to scrape various user data from a JSON API, og then store those values in a MySQL database I've set up.

The code seems to run fine for the most part, but some users does not have the attributes I'm trying to scrape in the JSON, and thus I'm left with Nonetype errors that I cant seem to foil.

If possible I'd like to just store "0" in the database where the json does not contain the attribute.

In the m/snippet below this works fine for users that has a job, but users without a job returns Nonetype on jobposition and apparently breaks the loop.

response = requests.get("URL")
json_obj = json.loads(response.text)

timer = json_obj['timestamp']
jobposition = json_obj['job']['position']

query = "INSERT INTO users (timer, jobposition) VALUES (%s, %s)"
values = (timer, jobposition)

cursor = db.cursor()
cursor.execute(query, values)
db.commit()

Thanks in advance!

python json
3 Answers

You can use for that the get() method of the dictionary as follow

timer = json_obj.get('timestamp', 0)

0 is the default value and in case there is no 'timestamp' attribute it will return 0. For job position, you can do

jobposition = json_obj['job'].get('position', 0) if 'job' in json_obj else 0
user_131012390

Try this

try:
 jobposition = json_obj['job']['position']
except:
 jobposition = 0
user_92355050

You can more clearly declare the data schema using dataclasses:

from dataclasses import dataclass

from validated_dc import ValidatedDC


@dataclass
class Job(ValidatedDC):
    name: str
    position: int = 0


@dataclass
class Workers(ValidatedDC):
    timer: int
    job: Job


input_data = {
    'timer': 123,
    'job': {'name': 'driver'}
}

workers = Workers(**input_data)

assert workers.job.position == 0

https://github.com/EvgeniyBurdin/validated_dc

user_53534840
Related