I have the following JSON structure obtained from here:
[{'country': 'Afghanistan',
'country_code': 'AFG',
'continent': 'Asia',
'population': 38928341,
'indicator': 'cases',
'weekly_count': 0,
'year_week': '2020-01',
'rate_14_day': '0',
'cumulative_count': 0,
'source': 'Epidemic intelligence national data'},
{'country': 'Afghanistan',
'country_code': 'AFG',
'continent': 'Asia',
'population': 38928341,
'indicator': 'cases',
'weekly_count': 0,
'year_week': '2020-02',
'rate_14_day': '0',
'cumulative_count': 0,
'source': 'Epidemic intelligence national data'},
...}
So I'm passing it to a transform task where the fields are separated in a list of lists to be transformed in a dataframe at the end of the process.
My problem is when some field is not present in a record, for example, in the case below where the field weekly_count is not present and the code brokes when comes to it:
{'country': 'Afghanistan',
'country_code': 'AFG',
'continent': 'Asia',
'population': 38928341,
'indicator': 'cases',
'year_week': '2021-52',
'rate_14_day': '0.922207293652714',
'cumulative_count': 158156,
'source': 'Epidemic intelligence national data'}
My code to pass from JSON to list is this:
def transform(data: dict) -> pd.DataFrame:
transformed = []
for record in data:
transformed.append({
'country': record['country'],
'country_code': record['country_code'],
'continent': record['continent'],
'population': record['population'],
'indicator': record['indicator'],
'weekly_count': record['weekly_count'],
'year_week': record['year_week'],
'cumulative_count': record['cumulative_count'],
'rate_14_day': record['rate_14_day'],
'source': record['source']
})
return pd.DataFrame(transformed)