I want to merge two dictionaries into a new dictionary.
x = {'a': 1, 'b': 2}
y = {'b': 3, 'c': 4}
z = merge(x, y)
>>> z
{'a': 1, 'b': 3, 'c': 4}
Whenever a key k is present in both dictionaries, only the value y[k] should be kept.
I want to merge two dictionaries into a new dictionary.
x = {'a': 1, 'b': 2}
y = {'b': 3, 'c': 4}
z = merge(x, y)
>>> z
{'a': 1, 'b': 3, 'c': 4}
Whenever a key k is present in both dictionaries, only the value y[k] should be kept.
In your case, what you can do is:
z = dict(list(x.items()) + list(y.items()))
This will, as you want it, put the final dict in z, and make the value for key b be properly overridden by the second (y) dict's value:
>>> x = {'a':1, 'b': 2}
>>> y = {'b':10, 'c': 11}
>>> z = dict(list(x.items()) + list(y.items()))
>>> z
{'a': 1, 'c': 11, 'b': 10}
If you use Python 2, you can even remove the list() calls. To create z:
>>> z = dict(x.items() + y.items())
>>> z
{'a': 1, 'c': 11, 'b': 10}
If you use Python version 3.9.0a4 or greater, then you can directly use:
x = {'a':1, 'b': 2}
y = {'b':10, 'c': 11}
z = x | y
print(z)
{'a': 1, 'c': 11, 'b': 10}
Another, more concise, option:
z = dict(x, **y)
Note: this has become a popular answer, but it is important to point out that if y has any non-string keys, the fact that this works at all is an abuse of a CPython implementation detail, and it does not work in Python 3, or in PyPy, IronPython, or Jython. Also, Guido is not a fan. So I can't recommend this technique for forward-compatible or cross-implementation portable code, which really means it should be avoided entirely.
This probably won't be a popular answer, but you almost certainly do not want to do this. If you want a copy that's a merge, then use copy (or deepcopy, depending on what you want) and then update. The two lines of code are much more readable - more Pythonic - than the single line creation with .items() + .items(). Explicit is better than implicit.
In addition, when you use .items() (pre Python 3.0), you're creating a new list that contains the items from the dict. If your dictionaries are large, then that is quite a lot of overhead (two large lists that will be thrown away as soon as the merged dict is created). update() can work more efficiently, because it can run through the second dict item-by-item.
In terms of time:
>>> timeit.Timer("dict(x, **y)", "x = dict(zip(range(1000), range(1000)))\ny=dict(zip(range(1000,2000), range(1000,2000)))").timeit(100000)
15.52571702003479
>>> timeit.Timer("temp = x.copy()\ntemp.update(y)", "x = dict(zip(range(1000), range(1000)))\ny=dict(zip(range(1000,2000), range(1000,2000)))").timeit(100000)
15.694622993469238
>>> timeit.Timer("dict(x.items() + y.items())", "x = dict(zip(range(1000), range(1000)))\ny=dict(zip(range(1000,2000), range(1000,2000)))").timeit(100000)
41.484580039978027
IMO the tiny slowdown between the first two is worth it for the readability. In addition, keyword arguments for dictionary creation was only added in Python 2.3, whereas copy() and update() will work in older versions.
In a follow-up answer, you asked about the relative performance of these two alternatives:
z1 = dict(x.items() + y.items())
z2 = dict(x, **y)
On my machine, at least (a fairly ordinary x86_64 running Python 2.5.2), alternative z2 is not only shorter and simpler but also significantly faster. You can verify this for yourself using the timeit module that comes with Python.
Example 1: identical dictionaries mapping 20 consecutive integers to themselves:
% python -m timeit -s 'x=y=dict((i,i) for i in range(20))' 'z1=dict(x.items() + y.items())'
100000 loops, best of 3: 5.67 usec per loop
% python -m timeit -s 'x=y=dict((i,i) for i in range(20))' 'z2=dict(x, **y)'
100000 loops, best of 3: 1.53 usec per loop
z2 wins by a factor of 3.5 or so. Different dictionaries seem to yield quite different results, but z2 always seems to come out ahead. (If you get inconsistent results for the same test, try passing in -r with a number larger than the default 3.)
Example 2: non-overlapping dictionaries mapping 252 short strings to integers and vice versa:
% python -m timeit -s 'from htmlentitydefs import codepoint2name as x, name2codepoint as y' 'z1=dict(x.items() + y.items())'
1000 loops, best of 3: 260 usec per loop
% python -m timeit -s 'from htmlentitydefs import codepoint2name as x, name2codepoint as y' 'z2=dict(x, **y)'
10000 loops, best of 3: 26.9 usec per loop
z2 wins by about a factor of 10. That's a pretty big win in my book!
After comparing those two, I wondered if z1's poor performance could be attributed to the overhead of constructing the two item lists, which in turn led me to wonder if this variation might work better:
from itertools import chain
z3 = dict(chain(x.iteritems(), y.iteritems()))
A few quick tests, e.g.
% python -m timeit -s 'from itertools import chain; from htmlentitydefs import codepoint2name as x, name2codepoint as y' 'z3=dict(chain(x.iteritems(), y.iteritems()))'
10000 loops, best of 3: 66 usec per loop
lead me to conclude that z3 is somewhat faster than z1, but not nearly as fast as z2. Definitely not worth all the extra typing.
This discussion is still missing something important, which is a performance comparison of these alternatives with the "obvious" way of merging two lists: using the update method. To try to keep things on an equal footing with the expressions, none of which modify x or y, I'm going to make a copy of x instead of modifying it in-place, as follows:
z0 = dict(x)
z0.update(y)
A typical result:
% python -m timeit -s 'from htmlentitydefs import codepoint2name as x, name2codepoint as y' 'z0=dict(x); z0.update(y)'
10000 loops, best of 3: 26.9 usec per loop
In other words, z0 and z2 seem to have essentially identical performance. Do you think this might be a coincidence? I don't....
In fact, I'd go so far as to claim that it's impossible for pure Python code to do any better than this. And if you can do significantly better in a C extension module, I imagine the Python folks might well be interested in incorporating your code (or a variation on your approach) into the Python core. Python uses dict in lots of places; optimizing its operations is a big deal.
You could also write this as
z0 = x.copy()
z0.update(y)
as Tony does, but (not surprisingly) the difference in notation turns out not to have any measurable effect on performance. Use whichever looks right to you. Of course, he's absolutely correct to point out that the two-statement version is much easier to understand.
I wanted something similar, but with the ability to specify how the values on duplicate keys were merged, so I hacked this out (but did not heavily test it). Obviously this is not a single expression, but it is a single function call.
def merge(d1, d2, merge_fn=lambda x,y:y):
"""
Merges two dictionaries, non-destructively, combining
values on duplicate keys as defined by the optional merge
function. The default behavior replaces the values in d1
with corresponding values in d2. (There is no other generally
applicable merge strategy, but often you'll have homogeneous
types in your dicts, so specifying a merge technique can be
valuable.)
Examples:
>>> d1
{'a': 1, 'c': 3, 'b': 2}
>>> merge(d1, d1)
{'a': 1, 'c': 3, 'b': 2}
>>> merge(d1, d1, lambda x,y: x+y)
{'a': 2, 'c': 6, 'b': 4}
"""
result = dict(d1)
for k,v in d2.iteritems():
if k in result:
result[k] = merge_fn(result[k], v)
else:
result[k] = v
return result
x = {'a':1, 'b': 2}
y = {'b':10, 'c': 11}
z = dict(x.items() + y.items())
print z
For items with keys in both dictionaries ('b'), you can control which one ends up in the output by putting that one last.
I benchmarked the suggested with perfplot and found that the good old
temp = x.copy()
temp.update(y)
is the fastest solution together with the new (Python 3.9+)
x | y
Code to reproduce the plot:
from collections import ChainMap
from itertools import chain
import perfplot
def setup(n):
x = dict(zip(range(n), range(n)))
y = dict(zip(range(n, 2 * n), range(n, 2 * n)))
return x, y
def copy_update(x, y):
temp = x.copy()
temp.update(y)
return temp
def add_items(x, y):
return dict(list(x.items()) + list(y.items()))
def curly_star(x, y):
return {**x, **y}
def chain_map(x, y):
return dict(ChainMap({}, y, x))
def itertools_chain(x, y):
return dict(chain(x.items(), y.items()))
def python39_concat(x, y):
return x | y
b = perfplot.bench(
setup=setup,
kernels=[
copy_update,
add_items,
curly_star,
chain_map,
itertools_chain,
python39_concat,
],
labels=[
"copy_update",
"dict(list(x.items()) + list(y.items()))",
"{**x, **y}",
"chain_map",
"itertools.chain",
"x | y",
],
n_range=[2 ** k for k in range(18)],
xlabel="len(x), len(y)",
equality_check=None,
)
b.save("out.png")
b.show()
If you don't mind mutating x,
x.update(y) or x
Simple, readable, performant. You know update() always returns None, which is a false value. So the above expression will always evaluate to x, after updating it.
Most mutating methods in the standard library (like .update()) return None by convention, so this kind of pattern will work on those too. However, if you're using a dict subclass or some other method that doesn't follow this convention, then or may return its left operand, which may not be what you want. Instead, you can use a tuple display and index, which works regardless of what the first element evaluates to (although it's not quite as pretty):
(x.update(y), x)[-1]
If you don't have x in a variable yet, you can use lambda to make a local without using an assignment statement. This amounts to using lambda as a let expression, which is a common technique in functional languages, but is maybe unpythonic.
(lambda x: x.update(y) or x)({'a': 1, 'b': 2})
Although it's not that different from the following use of the new walrus operator (Python 3.8+ only),
(x := {'a': 1, 'b': 2}).update(y) or x
especially if you use a default argument:
(lambda x={'a': 1, 'b': 2}: x.update(y) or x)()
If you do want a copy, PEP 584 style x | y is the most Pythonic on 3.9+. If you must support older versions, PEP 448 style {**x, **y} is easiest for 3.5+. But if that's not available in your (even older) Python version, the let expression pattern works here too.
(lambda z=x.copy(): z.update(y) or z)()
(That is, of course, nearly equivalent to (z := x.copy()).update(y) or z, but if your Python version is new enough for that, then the PEP 448 style will be available.)
New in Python 3.9: Use the union operator (|) to merge dicts similar to sets:
>>> d = {'a': 1, 'b': 2}
>>> e = {'a': 9, 'c': 3}
>>> d | e
{'a': 9, 'b': 2, 'c': 3}
For matching keys, the right dict takes precedence.
This also works for |= to modify a dict in-place:
>>> e |= d # e = e | d
>>> e
{'a': 1, 'c': 3, 'b': 2}
The problem I have with solutions listed to date is that, in the merged dictionary, the value for key "b" is 10 but, to my way of thinking, it should be 12. In that light, I present the following:
import timeit
n=100000
su = """
x = {'a':1, 'b': 2}
y = {'b':10, 'c': 11}
"""
def timeMerge(f,su,niter):
print "{:4f} sec for: {:30s}".format(timeit.Timer(f,setup=su).timeit(n),f)
timeMerge("dict(x, **y)",su,n)
timeMerge("x.update(y)",su,n)
timeMerge("dict(x.items() + y.items())",su,n)
timeMerge("for k in y.keys(): x[k] = k in x and x[k]+y[k] or y[k] ",su,n)
#confirm for loop adds b entries together
x = {'a':1, 'b': 2}
y = {'b':10, 'c': 11}
for k in y.keys(): x[k] = k in x and x[k]+y[k] or y[k]
print "confirm b elements are added:",x
0.049465 sec for: dict(x, **y)
0.033729 sec for: x.update(y)
0.150380 sec for: dict(x.items() + y.items())
0.083120 sec for: for k in y.keys(): x[k] = k in x and x[k]+y[k] or y[k]
confirm b elements are added: {'a': 1, 'c': 11, 'b': 12}
There will be a new option when Python 3.8 releases (scheduled for 20 October, 2019), thanks to PEP 572: Assignment Expressions. The new assignment expression operator := allows you to assign the result of the copy and still use it to call update, leaving the combined code a single expression, rather than two statements, changing:
newdict = dict1.copy()
newdict.update(dict2)
to:
(newdict := dict1.copy()).update(dict2)
while behaving identically in every way. If you must also return the resulting dict (you asked for an expression returning the dict; the above creates and assigns to newdict, but doesn't return it, so you couldn't use it to pass an argument to a function as is, a la myfunc((newdict := dict1.copy()).update(dict2))), then just add or newdict to the end (since update returns None, which is falsy, it will then evaluate and return newdict as the result of the expression):
(newdict := dict1.copy()).update(dict2) or newdict
Important caveat: In general, I'd discourage this approach in favor of:
newdict = {**dict1, **dict2}
The unpacking approach is clearer (to anyone who knows about generalized unpacking in the first place, which you should), doesn't require a name for the result at all (so it's much more concise when constructing a temporary that is immediately passed to a function or included in a list/tuple literal or the like), and is almost certainly faster as well, being (on CPython) roughly equivalent to:
newdict = {}
newdict.update(dict1)
newdict.update(dict2)
but done at the C layer, using the concrete dict API, so no dynamic method lookup/binding or function call dispatch overhead is involved (where (newdict := dict1.copy()).update(dict2) is unavoidably identical to the original two-liner in behavior, performing the work in discrete steps, with dynamic lookup/binding/invocation of methods.
It's also more extensible, as merging three dicts is obvious:
newdict = {**dict1, **dict2, **dict3}
where using assignment expressions won't scale like that; the closest you could get would be:
(newdict := dict1.copy()).update(dict2), newdict.update(dict3)
or without the temporary tuple of Nones, but with truthiness testing of each None result:
(newdict := dict1.copy()).update(dict2) or newdict.update(dict3)
either of which is obviously much uglier, and includes further inefficiencies (either a wasted temporary tuple of Nones for comma separation, or pointless truthiness testing of each update's None return for or separation).
The only real advantage to the assignment expression approach occurs if:
sets and dicts (both of them support copy and update, so the code works roughly as you'd expect it to)dict itself, and must preserve the type and semantics of the left hand side (rather than ending up with a plain dict). While myspecialdict({**speciala, **specialb}) might work, it would involve an extra temporary dict, and if myspecialdict has features plain dict can't preserve (e.g. regular dicts now preserve order based on the first appearance of a key, and value based on the last appearance of a key; you might want one that preserves order based on the last appearance of a key so updating a value also moves it to the end), then the semantics would be wrong. Since the assignment expression version uses the named methods (which are presumably overloaded to behave appropriately), it never creates a dict at all (unless dict1 was already a dict), preserving the original type (and original type's semantics), all while avoiding any temporaries.>>> x = {'a':1, 'b': 2}
>>> y = {'b':10, 'c': 11}
>>> x, z = dict(x), x.update(y) or x
>>> x
{'a': 1, 'b': 2}
>>> y
{'c': 11, 'b': 10}
>>> z
{'a': 1, 'c': 11, 'b': 10}
Python 3.9+ only
Merge (|) and update (|=) operators have been added to the built-in dict class.
>>> d = {'spam': 1, 'eggs': 2, 'cheese': 3}
>>> e = {'cheese': 'cheddar', 'aardvark': 'Ethel'}
>>> d | e
{'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
The augmented assignment version operates in-place:
>>> d |= e
>>> d
{'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
See PEP 584
In Python 3.9
Based on PEP 584, the new version of Python introduces two new operators for dictionaries: union (|) and in-place union (|=). You can use | to merge two dictionaries, while |= will update a dictionary in place:
>>> pycon = {2016: "Portland", 2018: "Cleveland"}
>>> europython = {2017: "Rimini", 2018: "Edinburgh", 2019: "Basel"}
>>> pycon | europython
{2016: 'Portland', 2018: 'Edinburgh', 2017: 'Rimini', 2019: 'Basel'}
>>> pycon |= europython
>>> pycon
{2016: 'Portland', 2018: 'Edinburgh', 2017: 'Rimini', 2019: 'Basel'}
If d1 and d2 are two dictionaries, then d1 | d2 does the same as {**d1, **d2}. The | operator is used for calculating the union of sets, so the notation may already be familiar to you.
One advantage of using | is that it works on different dictionary-like types and keeps the type through the merge:
>>> from collections import defaultdict
>>> europe = defaultdict(lambda: "", {"Norway": "Oslo", "Spain": "Madrid"})
>>> africa = defaultdict(lambda: "", {"Egypt": "Cairo", "Zimbabwe": "Harare"})
>>> europe | africa
defaultdict(<function <lambda> at 0x7f0cb42a6700>,
{'Norway': 'Oslo', 'Spain': 'Madrid', 'Egypt': 'Cairo', 'Zimbabwe': 'Harare'})
>>> {**europe, **africa}
{'Norway': 'Oslo', 'Spain': 'Madrid', 'Egypt': 'Cairo', 'Zimbabwe': 'Harare'}
You can use a defaultdict when you want to effectively handle missing keys. Note that | preserves the defaultdict, while {**europe, **africa} does not.
There are some similarities between how | works for dictionaries and how + works for lists. In fact, the + operator was originally proposed to merge dictionaries as well. This correspondence becomes even more evident when you look at the in-place operator.
The basic use of |= is to update a dictionary in place, similar to .update():
>>> libraries = {
... "collections": "Container datatypes",
... "math": "Mathematical functions",
... }
>>> libraries |= {"zoneinfo": "IANA time zone support"}
>>> libraries
{'collections': 'Container datatypes', 'math': 'Mathematical functions',
'zoneinfo': 'IANA time zone support'}
When you merge dictionaries with |, both dictionaries need to be of a proper dictionary type. On the other hand, the in-place operator (|=) is happy to work with any dictionary-like data structure:
>>> libraries |= [("graphlib", "Functionality for graph-like structures")]
>>> libraries
{'collections': 'Container datatypes', 'math': 'Mathematical functions',
'zoneinfo': 'IANA time zone support',
'graphlib': 'Functionality for graph-like structures'}
Deep merge of dicts:
from typing import List, Dict
from copy import deepcopy
def merge_dicts(*from_dicts: List[Dict], no_copy: bool=False) -> Dict :
""" no recursion deep merge of 2 dicts
By default creates fresh Dict and merges all to it.
no_copy = True, will merge all dicts to a fist one in a list without copy.
Why? Sometime I need to combine one dictionary from "layers".
The "layers" are not in use and dropped immediately after merging.
"""
if no_copy:
xerox = lambda x:x
else:
xerox = deepcopy
result = xerox(from_dicts[0])
for _from in from_dicts[1:]:
merge_queue = [(result, _from)]
for _to, _from in merge_queue:
for k, v in _from.items():
if k in _to and isinstance(_to[k], dict) and isinstance(v, dict):
# key collision add both are dicts.
# add to merging queue
merge_queue.append((_to[k], v))
continue
_to[k] = xerox(v)
return result
usage:
print("=============================")
print("merge all dicts to first one without copy.")
a0 = {"a":{"b":1}}
a1 = {"a":{"c":{"d":4}}}
a2 = {"a":{"c":{"f":5}, "d": 6}}
print(f"a0 id[{id(a0)}] value:{a0}")
print(f"a1 id[{id(a1)}] value:{a1}")
print(f"a2 id[{id(a2)}] value:{a2}")
r = merge_dicts(a0, a1, a2, no_copy=True)
print(f"r id[{id(r)}] value:{r}")
print("=============================")
print("create fresh copy of all")
a0 = {"a":{"b":1}}
a1 = {"a":{"c":{"d":4}}}
a2 = {"a":{"c":{"f":5}, "d": 6}}
print(f"a0 id[{id(a0)}] value:{a0}")
print(f"a1 id[{id(a1)}] value:{a1}")
print(f"a2 id[{id(a2)}] value:{a2}")
r = merge_dicts(a0, a1, a2)
print(f"r id[{id(r)}] value:{r}")
I think my ugly one-liners are just necessary here.
z = next(z.update(y) or z for z in [x.copy()])
# or
z = (lambda z: z.update(y) or z)(x.copy())
P.S. This is a solution working in both versions of Python. I know that Python 3 has this {**x, **y} thing and it is the right thing to use (as well as moving to Python 3 if you still have Python 2 is the right thing to do).
I didn't see any methods yet for deep merging (I will happily be corrected though). Making use of the | operator in 3.9+ for the use case of dict new being a set of default settings, and dict existing being a set of existing settings in use. My goal was to merge in any added settings from new without over writing existing settings in existing. I believe this recursive implementation will allow one to upgrade a dict with new values from another dict.
Comments are welcome, as this is one of my first real novel contributions to SO. I'd happily be corrected if the code is not up to snuff.
def merge_dict_recursive(new: dict, existing: dict):
merged = new | existing
for k, v in merged.items():
if isinstance(v, dict):
if k not in existing:
# key is not in existing dict at all, so add entire value
existing[k] = new[k]
merged[k] = merge_dict_recursive(new[k], existing[k])
return merged
Example test data:
new
{'dashboard': True,
'depth': {'a': 1, 'b': 22222, 'c': {'d': {'e': 69}}},
'intro': 'this is the dashboard',
'newkey': False,
'show_closed_sessions': False,
'version': None,
'visible_sessions_limit': 9999}
existing
{'dashboard': True,
'depth': {'a': 5},
'intro': 'this is the dashboard',
'newkey': True,
'show_closed_sessions': False,
'version': '2021-08-22 12:00:30.531038+00:00'}
merged
{'dashboard': True,
'depth': {'a': 5, 'b': 22222, 'c': {'d': {'e': 69}}},
'intro': 'this is the dashboard',
'newkey': True,
'show_closed_sessions': False,
'version': '2021-08-22 12:00:30.531038+00:00',
'visible_sessions_limit': 9999}