Modifying a Python dict while iterating over it

Viewed 100427

Let's say we have a Python dictionary d, and we're iterating over it like so:

for k, v in d.iteritems():
    del d[f(k)] # remove some item
    d[g(k)] = v # add a new item

(f and g are just some black-box transformations.)

In other words, we try to add/remove items to d while iterating over it using iteritems.

Is this well defined? Could you provide some references to support your answer?


See also How to avoid "RuntimeError: dictionary changed size during iteration" error? for the separate question of how to avoid the problem.

9 Answers

Python 3 you should just:

prefix = 'item_'
t = {'f1': 'ffw', 'f2': 'fca'}
t2 = dict() 
for k,v in t.items():
    t2[k] = prefix + v

or use:

t2 = t1.copy()

You should never modify original dictionary, it leads to confusion as well as potential bugs or RunTimeErrors. Unless you just append to the dictionary with new key names.

This question asks about using an iterator (and funny enough, that Python 2 .iteritems iterator is no longer supported in Python 3) to delete or add items, and it must have a No as its only right answer as you can find it in the accepted answer. Yet: most of the searchers try to find a solution, they will not care how this is done technically, be it an iterator or a recursion, and there is a solution for the problem:

You cannot loop-change a dict without using an additional (recursive) function.

This question should therefore be linked to a question that has a working solution:

By the same recursive methods, you will also able to add items as the question asks for as well.


Since my request to link this question was declined, here is a copy of the solution that can delete items from a dict. See How can I remove a key:value pair wherever the chosen key occurs in a deeply nested dictionary? (= "delete") for examples / credits / notes.

import copy

def find_remove(this_dict, target_key, bln_overwrite_dict=False):
    if not bln_overwrite_dict:
        this_dict = copy.deepcopy(this_dict)

    for key in this_dict:
        # if the current value is a dict, dive into it
        if isinstance(this_dict[key], dict):
            if target_key in this_dict[key]:
                this_dict[key].pop(target_key)

            this_dict[key] = find_remove(this_dict[key], target_key)

    return this_dict

dict_nested_new = find_remove(nested_dict, "sub_key2a")

The trick

The trick is to find out in advance whether a target_key is among the next children (= this_dict[key] = the values of the current dict iteration) before you reach the child level recursively. Only then you can still delete a key:value pair of the child level while iterating over a dictionary. Once you have reached the same level as the key to be deleted and then try to delete it from there, you would get the error:

RuntimeError: dictionary changed size during iteration

The recursive solution makes any change only on the next values' sub-level and therefore avoids the error.

Today I had a similar use-case, but instead of simply materializing the keys on the dictionary at the beginning of the loop, I wanted changes to the dict to affect the iteration of the dict, which was an ordered dict.

I ended up building the following routine, which can also be found in jaraco.itertools:

def _mutable_iter(dict):
    """
    Iterate over items in the dict, yielding the first one, but allowing
    it to be mutated during the process.
    >>> d = dict(a=1)
    >>> it = _mutable_iter(d)
    >>> next(it)
    ('a', 1)
    >>> d
    {}
    >>> d.update(b=2)
    >>> list(it)
    [('b', 2)]
    """
    while dict:
        prev_key = next(iter(dict))
        yield prev_key, dict.pop(prev_key)

The docstring illustrates the usage. This function could be used in place of d.iteritems() above to have the desired effect.

Related