How to connect json files downloaded from api?

user_13134420 0
Viewed 69

I am downloading hundreds of files which have a format:

{
    "result": [
        {
            "Lines": "130",
            "Lon": 21.0566243,
            "VehicleNumber": "1000",
            "Time": "2020-12-22 18:55:03",
            "Lat": 52.1812215,
            "Brigade": "1"
        },
        {
            "Lines": "311",
            "Lon": 21.0817553,
            "VehicleNumber": "1001",
            "Time": "2020-12-22 18:54:52",
            "Lat": 52.2407755,
            "Brigade": "2"
        }
    ]
}

My desired output is a list of dictionaries

[
    {
        "Lines": "130",
        "Lon": 21.0566243,
        "VehicleNumber": "1000",
        "Time": "2020-12-22 18:55:03",
        "Lat": 52.1812215,
        "Brigade": "1"
    },
    {
        "Lines": "311",
        "Lon": 21.0817553,
        "VehicleNumber": "1001",
        "Time": "2020-12-22 18:54:52",
        "Lat": 52.2407755,
        "Brigade": "2"
    }
]

combined from all the files.

What is a proper way to handle it?

I tried downloading with

def download(file_name):
    with open(os.path.join(path_to_data,file_name), 'a') as outfile:
        json.dump(response.json(), outfile)

But then I got one file with a couple of dictionaries with {"result":} and can't even load it as a json. Should I save each json in a separate file instead of making it just one file? If so, should i make a list of names for function download?

python json
1 Answers

It's not clear if you want each response to be a list of the dictionaries or if you want one big list written to the file.

You can collect all dictionaries just by creating a list and using .extend. That's one large list with dictionaries.

hold_list = []
# your API loop here
    resp = response.json()
    hold_list.extend(resp['result'])
print(hold_list)

If you want a list of lists, use .append instead of .extend. Play around with it to see the difference.

After that, you can dump it into a file as a JSON:

with open("output.json", "w") as fp:
    json.dump(hold_list, fp)

Lastly, if you want to write to the file each time you get the response from the API, you can write resp['result']. But that gives a list for each API response, and you'll need to either write a delimiter or put in a new line character or you may end up with a list after list with no spaces or delimiters in-between. This won't be JSON, but you can use Python and manipulate it as a list with dictionaries.

However, it is possible to get a JSON as well. For example, like this (gets a list of lists, like .append in the first case):

with open("output.json", "w") as fp:
    fp.write("[")
    first = True
    # your API loop here
        if first:
            first = False
        else:
            fp.write(", ")
        fp.write(json.dumps(response_json["result"]))
    fp.write("]")

OR (a list of dicts, like .extend): replace this line:

fp.write(json.dumps(response_json["result"]))

with this one:

fp.write(json.dumps(response_json["result"])[1:-1])
# [1:-1] is a slice to remove the [ and ]
user_121334340
Related