Python version 3.7 - 3.9 pandas 1.4.4
>>> import pandas as pd
>>> import numpy as np
>>> df1 = df1 = pd.DataFrame({'name':[np.nan, np.nan, 'John', 'Mary', 'David'],
... 'age' :[np.nan, np.nan, 20, 30, 40],
... 'state': ['NY', 'CA', 'CA', 'NY', 'IL']})
>>> df1
name age state
0 NaN NaN NY
1 NaN NaN CA
2 John 20.0 CA
3 Mary 30.0 NY
4 David 40.0 IL
>>> df2 = pd.DataFrame({'name':['Lee', 'David', 'Mary', np.nan],
... 'age':[np.nan, 40, 30, np.nan],
... 'city':['Boston', 'Chicago', 'New York', 'Seattle']})
>>> df2
name age city
0 Lee NaN Boston
1 David 40.0 Chicago
2 Mary 30.0 New York
3 NaN NaN Seattle
>>> df1_new = df1.set_index(['name', 'age'])
>>> df2_new = df2.set_index(['name', 'age'])
>>> df1_new.index
MultiIndex([( nan, nan),
( nan, nan),
( 'John', 20.0),
( 'Mary', 30.0),
('David', 40.0)],
names=['name', 'age'])
>>> df2_new.index
MultiIndex([( 'Lee', nan),
('David', 40.0),
( 'Mary', 30.0),
( nan, nan)],
names=['name', 'age'])
>>> df1_new.index.union(df2_new.index)
MultiIndex([('David', 40.0),
( 'John', 20.0),
( 'Mary', 30.0),
( nan, nan),
( nan, nan)],
names=['name', 'age'])
>>> df2_new.index.union(df1_new.index)
MultiIndex([('David', 40.0),
( 'John', 20.0),
( 'Lee', nan),
( 'Mary', 30.0),
( nan, nan),
( nan, nan)],
names=['name', 'age'])
So the output of df1_new.index.union(df2_new.index) has a missing key ( 'Lee', nan) that is in df2_new.index.union(df1_new.index). It is an unexpected behavior as mathematically a union operation shall be not dependent on the sequence of operands.
I notice this issue when I am trying to figure out how to ensure `pandas.merge(df1, df2, left_index=True, right_index=True, how='outer') will have all indices included.