How can the number of times a key has values stored in it it be counted in a nested dictionary?

jefazo92 0
Viewed 27

I'm trying to count the number of times a key contains information (i.e. a list of dictionaries) amongst multiple files. I have around 200 JSON files and each of them have a "network" dictionary with other nested dictionaries corresponding to the protocols captured (e.g. if nested dictionary key tcp is empty then that protocol hasn't been used). I'd want to implement the count using Counters(). One section of my JSON file looks something like this:

"network": {
        "tls": [], 
        "udp": [
            {
                "src": "192.168.56.101", 
                "time": 5.9932849407196045, 
                "offset": 1532, 
                "dport": 137, 
                "npackets": 108, 
                "dst": "192.168.56.255", 
                "sport": 137, 
                "nbytes": 7128
            }, 
            {
                "src": "192.168.56.101", 
                "time": 11.992594003677368, 
                "offset": 14060, 
                "dport": 138, 
                "npackets": 28, 
                "dst": "192.168.56.255", 
                "sport": 138, 
                "nbytes": 5588
            }, 
            {
                "src": "192.168.56.101", 
                "time": 3.9214370250701904, 
                "offset": 21904, 
                "dport": 5355, 
                "npackets": 4, 
                "dst": "224.0.0.252", 
                "sport": 49674, 
                "nbytes": 144
            }, 
            {
                "src": "192.168.56.101", 
                "time": 5.920687913894653, 
                "offset": 22248, 
                "dport": 5355, 
                "npackets": 4, 
                "dst": "224.0.0.252", 
                "sport": 50699, 
                "nbytes": 144
            }, 
            {
                "src": "192.168.56.101", 
                "time": 3.1062138080596924, 
                "offset": 22592, 
                "dport": 5355, 
                "npackets": 4, 
                "dst": "224.0.0.252", 
                "sport": 53207, 
                "nbytes": 128
            },

I want to count the number of times key "udp" has a list of dictionary stored within it amongst 100-200 fles (e.g. 150 files show UDP traffic. I also want to merge together the counts of keys "udp", "tcp", "icmp", "http", etc. into a single data structure whose histogram I can then plot.

So far I've tried to store every occurrence in a set and pass it then to a list to apply and store it on a Counter() later on. However, since each entry contains a list of dictionaries, I can't store these in a set.

counter list python-3.x json python
1 Answers

I am assuming you know how to read files within a directory therefore I'm mocking that part using the files array.

Then you can just use a dictionary which keeps track of the count of connections.

# a file
file1 = {
    "network": {
        "tls": [],
        "udp": [1]
    }
}
# another file
file2 = {
    "network": {
        "tls": [1],
        "udp": [1, 2, 3],
        "icmp": [1, 2]
    }
}
# yet another file
file3 = {
    "network": {
        "tls": [1, 2, 3, 4, 5, 6],
        "udp": [],
        "icmp": []
    }
}
# you will have an iterable where you can iterate over all files 
# similar to this array. As soon as you have that use the following algorithm
files = [file1, file2, file3]

# list of protocols you want to know the number of connections
relevant_protocols = ["tls", "udp", "icmp"]
# initialize dictionary with count of 0 for every relevant protocol
counter_dict = {protocol: 0 for protocol in relevant_protocols}
# loop over all files
for file in files:
    connections = file["network"]
    # for each of the protocols you're interested in increase the count
    # by the number of entries within the connection dictionary for that 
    # protocol. Be sure to check if such that protocol exist in the
    # connections to avoid a KeyError
    for protocol in relevant_protocols:
        if protocol in connections:
            counter_dict[protocol] += len(connections[protocol])
print(counter_dict)

Expected output:

{'tls': 7, 'udp': 4, 'icmp': 2}
Mushroomator0
Related