I'm working with a large number of large json files. I've written the below (extremely un elegant code) in order to generate two dictionaries with which I can create a dataframe to work with. However, in instances where the JSON has values with empty arrays, my code is propagating the last 'valid' values into the subsequent objects with empty arrays. I've tried replacing empty arrays with blanks but that doesn't seem to work either. (I know my code is very bad - still learning so please keep that in mind)
dicts = []
fid = []
x=0
while x < 1:
for i in files:
n=[]
k = []
t = []
op = open(i)
data = op.read()
js = json.loads(data)
items = js['metadata']['items']
#items = json.dumps(items).replace('[]','""')
#items = json.loads(items)
fileid = js['id']
fid.append(fileid)
##Everything after this point is what's throwing me off##
for a in items:
b = json.loads(json.dumps(a, sort_keys =True))
key = b['name']
k.append(key)
val = b['values']
values = []
for c in val:
j=json.dumps(c['value'])
if isinstance(c, list) == False:
continue
values.append(j)
j = ';'.join(values) #<-- For objects with more than one value
t.append(j)
output_dict = dict(zip([key], [j]))
n.append(output_dict)
dicts.append(n)
x = x+1
Here is an example section of the json where I'm observing this behavior:
x = {"metadata": {
"items": [
{
"values": [
{ "attribute1": "attribute", #<-- NOT IMPORTANT
"value": "VALUE 1" #<----VALUE I'M AFTER
},
{"attribute2": "attribute",#<-- NOT IMPORTANT
"value2": "VALUE 2"#<----VALUE I'M AFTER
}
],
"name": "NAME 1" #<--NAME I'M AFTER
},
{
"values": [
{
"value": []#<-- EMPTY ARRAY
}
],
"name": "NAME 2"}
]
}
}
In the above snippet, my ideal output is a list of dictionary pairings that looks like:
[{"NAME 1": "VALUE 1; VALUE 2", "NAME 2": " "...}]
But what I'm getting is:
[{"NAME 1": "VALUE 1; VALUE 2"}, {"NAME 2": "VALUE 1; VALUE 2"}...}]
I've tried deconstructing my work, and can't figure out why. I've re-indented and done a walk through a couple times and I don't understand why it would behave like this. What about the way my loop is constructed is causing this?