python multi threading damaged with use of loops

Viewed 32

I am using ThreadPoolExecutor module in Python but I have a problem when executing my script.

Here what I want to do:

I have this json file, called result:


{
    "hostname1": {
        "leaf_ip": "10.234.12.4",
        "hosts": [
            {
                "ip": "10.42.78.59",
                "mac": "ab:cd:ef:gh:ij:01",
                "vlan": "107"
            },
            {
                "ip": "10.42.78.60",
                "mac": "ab:cd:ef:gh:ij:02",
                "vlan": "161"
            },
            {
                "ip": "10.42.78.61",
                "mac": "ab:cd:ef:gh:ij:03",
                "vlan": "161"
            }
        ]
    },
    "hostname2": {
        "leaf_ip": "10.234.12.5",
        "hosts": [
            {
                "ip": "10.42.78.62",
                "mac": "ab:cd:ef:gh:ij:04",
                "vlan": "107"
            },
            {
                "ip": "10.42.78.64",
                "mac": "ab:cd:ef:gh:ij:06",
                "vlan": "161"
            }
        ]
    },
    "hostname3": {
        "leaf_ip": "10.234.12.6",
        "hosts": [
            {
                "ip": "10.42.78.65",
                "mac": "ab:cd:ef:gh:ij:07",
                "vlan": "107"
            },
            {
                "ip": "10.42.78.67",
                "mac": "ab:cd:ef:gh:ij:09",
                "vlan": "161"
            }
        ]
    },
    "hostname4": {
        "leaf_ip": "10.234.12.6",
        "hosts": []
    }
}

And I want to fill it by adding the "port" in each host like that:

"ip": "10.42.78.59",
"mac": "ab:cd:ef:gh:ij:01",
"vlan": "107",
"port": "port1"

In order to get the port, I am contacting the leaf (leaf_ip in the json file) and I'm launching a command on the switch, then I get the result that I am filtering.

Here is the whole code:

def main_loop(result):
    with ThreadPoolExecutor(max_workers=os.cpu_count() + 4) as executor:
        for hostname in result.keys():
            if result[hostname]["hosts"] != []:
                leaf_ip = result[hostname]["leaf_ip"]
                gateway_session = SSHSession('10.11.121.12','admin', password='******').open()
                remote_session = gateway_session.get_remote_session(leaf_ip, 'admin', password='*********')
                for host in result[hostname]["hosts"]:
                    i = result[hostname]["hosts"].index(host)
                    executor.submit(get_port, hostname, i, remote_session, result)
    return result
def get_port(hostname, i, remote_session, result):
    mac = result[hostname]["hosts"][i]["mac"]
    leaf_command = 'sh mac address-table address ' + mac + ' | json'
    process = remote_session.get_cmd_output(leaf_command)
    if process != b'':
        if type(json.loads(process)["TABLE_mac_address"]["ROW_mac_address"]) is dict:
            result[hostname]["hosts"][i]["port"] = json.loads(process)["TABLE_mac_address"]["ROW_mac_address"]["disp_port"]
        elif type(json.loads(process)["TABLE_mac_address"]["ROW_mac_address"]) is list:
            result[hostname]["hosts"][i]["port"] = json.loads(process)["TABLE_mac_address"]["ROW_mac_address"][0]["disp_port"]

So to sum up, for each hostname in the json, then for each host in the list of hosts if it is not empty, I connect with the leaf_ip corresponding to the hostname, and I will retrieve data on the switch that I add to my json after filtering.

After having launched my script, I notice that the elements of the beginning do not receive the port whereas those of the end yes. There seems to be a problem with multithreading because it works fine without it. I specify that the json is actually much longer with hundreds of elements, hence the need for multithreading.

I feel like I used multithreading wrong and would like to know if anyone has any idea what my mistake was.

Thank you

1 Answers

In main_loop you are iterating through items when you basically should iterate the keys:

for hostname in result.keys():
Related