Quickly mapping/modifying values in a large list of Python dicts?

user_1143387 0
Viewed 82

I have some code that I'm trying to speed up. Maybe what I've got is right, but whenever I ask on StackOverflow somebody usually knows a clever little trick "Use map!", "try this lambda", or "import iteratetools" and I'm hoping somebody can help here. This is the section of code I'm concerned with:

#slowest part from here....
for row_dict in json_data:
    row_dict_clean = {}
    for key, value in row_dict.items():
        value_clean = get_cleantext(value)
        row_dict_clean[key] = value_clean
    json_data_clean.append(row_dict_clean)
    total += 1
#to here...

The concept is pretty simple. I have a multi-million long list that contains dictionaries and I need to run each value through a little cleaner. Then I end up with a nice list of cleaned dictionaries. Any clever iterate tool that I'm not aware of that I should be using? Here is a more complete MVE to help play with it:

def get_json_data_clean(json_data):
    json_data_clean = []
    total = 0
    #slowest part from here....
    for row_dict in json_data:
        row_dict_clean = {}
        for key, value in row_dict.items():
            value_clean = get_cleantext(value)
            row_dict_clean[key] = value_clean
        json_data_clean.append(row_dict_clean)
        total += 1
    #to here...
    return json_data_clean

def get_cleantext(value):
    #do complex cleaning stuffs on the string, I can't change what this does
    value = value.replace("bad", "good")
    return value

json_data = [
    {"key1":"some bad",
     "key2":"bad things",
     "key3":"extra bad"},
    {"key1":"more bad stuff",
     "key2":"wow, so much bad",
     "key3":"who dis?"},
    # a few million more dictionaries
    {"key1":"so much bad stuff",
     "key2":"the bad",
     "key3":"the more bad"},
]

json_data_clean = get_json_data_clean(json_data)
print(json_data_clean)

Anytime I have nested for loops a little bell rings in my head, there is probably a better way to do this. Any help is appreciated!

python-3.x python performance
1 Answers

Must definitely ask clever guys at https://codereview.stackexchange.com/, but as a quick fix it appears you can just map() your transformation fucntion over a list of dictionaries as below:

def clean_text(value: str)-> str:
    # ...
    return value.replace("bad", "good")

def clean_dict(d: dict):
    return {k:clean_text(v) for k,v in d.items()}


json_data = [
    {"key1":"some bad",
     "key2":"bad things",
     "key3":"extra bad"},
    {"key1":"more bad stuff",
     "key2":"wow, so much bad",
     "key3":"who dis?"},
    # a few million more dictionaries
    {"key1":"so much bad stuff",
     "key2":"the bad",
     "key3":"the more bad"},
]

x = list(map(clean_dict, json_data))

A thing that gets left out is your total counter, but it never seems to leave the get_json_data_clean() anyways.

Not sure why @Daniel Gale proposed filter() as you are not throughing any values away, just transforming them.

user_17583630
Related