How to flatten the nested JSON file in order to retrieve an expected fields

user_11622712 0
Viewed 107

I have the following JSON file:

[ 
  {'docType': 'custom',
   'fields': 
    {
      'general_info': None,
      'power': 20,
      'safety': 
       {
         'boundingBox': [2.375,9.9,4.98,9.9,4.98,10.245,2.375,10.245],
         'confidence': 0.69,
         'page': 22,
         'text': 'bla-bla-bla',
         'type': 'string',
         'valueString': 'bla-bla-bla'
       },
       'replacement': 
        {
          'boundingBox': [2.505,2.51,2.54,2.51,2.54,3.425,2.505,3.425],
          'confidence': 0.262,
          'page': 7,
          'text': 'bla-bla-bla',
          'type': 'string',
          'valueString': 'bla-bla-bla'
         },
        'document_id': 'x123'
     } 
   }
]

I want to go through all field values and extract text from nested fields. The expected results is as the follows:

{
   'labels': 
    {
       'general_info': None,
       'power': 20,
       'safety': 'bla-bla-bla',
       'replacement': 'bla-bla-bla',
       'document_id': 'x123'
     } 
}

How can I flatted my JSON file and get an expected result?

This is what I have tried so far:

import json

json_object = json.load(raw_json)

fields = {}
for field in json_object:
    for attribute, value in field.items():
        fields[attribute] = value

fields_json = json.dumps(fields, indent = 4)

However, I don't know how to recursively enter into nested fields

python json
3 Answers

You can write a recursive function. It should call itself when a value is a dictionary.

This is an example.

def flatten_fields(d):
    out = {}
    for k, v in d.items():
        if isinstance(v, dict):
            out[k] = flatten_fields(v)
        elif k == 'text':
            return v
        elif isinstance(v, list):
            continue
        else:
            out[k] = v
    return out

To run it, you can iterate through each dictionary in the json_object. You only have one example above, but this is the how:

labels = []
for d in json_object:
    labels.append({'labels': flatten_fields(d.get('fields', {}))})

labels
# returns:
[{'labels': {'general_info': None,
   'power': 20,
   'safety': 'bla-bla-bla',
   'replacement': 'bla-bla-bla',
   'document_id': 'x123'}}]
user_50037560

After load it as python list just loop over it to get inside dict key called fields and simply loop on its keys and values once you found value whose type is dict you have to loop on it to and get the inside value whose key is text then get value only and the key be parent key

Example

from pprint import pprint

res = {}
for sub in content:
   for x, y in sub['fields'].items():
    if isinstance(y, dict):
        for i, e in y.items():
            if i == 'text':
                res[x] = e
    else:
        res[x] = y

final = {}
final['label'] = res
pprint(final)

output

{'label': {'document_id': 'x123',
           'general_info': None,
           'power': 20,
           'replacement': 'bla-bla-bla',
           'safety': 'bla-bla-bla'}}
user_135368930

you should use recursion to walk through dictionary. My solution would be:

import json
with open('raw_json', 'r') as j:
    d = json.load(j)
    # print(d)

def dict_walker(obj ,key=None):
    if isinstance(obj, dict):
        for key in obj:
            dict_walker(obj[key], key)
    else:
        print(key, ':', obj)

dict_walker(d)

OUT:

docType : custom
general_info : None
power : 20
boundingBox : [2.375, 9.9, 4.98, 9.9, 4.98, 10.245, 2.375, 10.245]
confidence : 0.69
page : 22
text : bla-bla-bla
type : string
valueString : bla-bla-bla
boundingBox : [2.505, 2.51, 2.54, 2.51, 2.54, 3.425, 2.505, 3.425]
confidence : 0.262
page : 7
text : bla-bla-bla
type : string
valueString : bla-bla-bla
document_id : x123
user_139655170
Related