Efficiently copying values from one ndarray to another on unequal sized arrays

user_9492900 0
Viewed 68

I have two arrays of different sizes, but I am trying to overwrite some values within the first array with values from the second array on the matching "keys". My actual problem may have many, many rows, and I have already determined that this is currently bottle-necking my program.

edit: I failed to recognize that there may be duplicate values in a1, which should stay duplicated. I added one such example to the np.array examples.

example:

import numpy as np

# first two columns are 'keys', overwrite the 3rd column in a1 with the 3rd column from a2
# some values may be missing from a2. Those should keep the value in a1

a1 = np.array([[ 0.0,  2.0,  10.0 ],
               [ 0.0,  2.0,  10.0 ],
               [ 0.0,  3.0,  10.0 ],
               [ 1.0,  3.0,  10.0 ],
               [ 1.0, 13.0,  10.0 ],
               [ 2.0,  2.0,  10.0 ],
               [ 2.0,  5.0,  10.0 ]])

a2 = np.array([[ 0.0,  2.0,  0.0   ],
               [ 0.0,  3.0,  0.713 ],
               [ 1.0,  3.0,  0.713 ],
               [ 1.0, 13.0,  1.0   ],
               [ 2.0,  2.0,  0.0   ]])

# wanted result:
np.array([[ 0.0,  2.0,  0.0   ],
          [ 0.0,  2.0,  0.0   ],
          [ 0.0,  3.0,  0.713 ],
          [ 1.0,  3.0,  0.713 ],
          [ 1.0, 13.0,  1.0   ],
          [ 2.0,  2.0,  0.0   ],
          [ 2.0,  5.0,  10.0   ]])

When I do this brute force, I would simply take each row in a2 and loop through each row in a1 to replace values on matches, but is there a way to do this that runs more efficiently? Some way to vectorize the operation on at least one of the loops? My actual case involves many rows in both arrays and this takes a looooong time.

numpy optimization python
3 Answers

Would you consider other packages like Pandas?

import pandas as pd

d2 = pd.DataFrame(a2).set_index([0,1])
d1 = pd.DataFrame(a1).set_index([0,1])

d1.update(d2)
d1.reset_index().values

Output:

array([[ 0.   ,  2.   ,  0.   ],
       [ 0.   ,  2.   ,  0.   ],
       [ 0.   ,  3.   ,  0.713],
       [ 1.   ,  3.   ,  0.713],
       [ 1.   , 13.   ,  1.   ],
       [ 2.   ,  2.   ,  0.   ],
       [ 2.   ,  5.   , 10.   ]])
user_42384080

Concatenate a2 and a1 and leave only unique rows for first 2 columns.

a_all = np.r_[a2, a1]
a_all = a_all[np.unique(a_all[:, :2], axis=0, return_index=True)[1]]
user_88525510

If column three is getting updated and you want to use pandas:

import numpy as np
import pandas as pd

a1 = np.array([[ 0.0,  2.0,  10.0 ],
               [ 0.0,  2.0,  10.0 ],
               [ 0.0,  3.0,  10.0 ],
               [ 1.0,  3.0,  10.0 ],
               [ 1.0, 13.0,  10.0 ],
               [ 2.0,  2.0,  10.0 ],
               [ 2.0,  5.0,  10.0 ]])

a2 = np.array([[ 0.0,  2.0,  0.0   ],
               [ 0.0,  3.0,  0.713 ],
               [ 1.0,  3.0,  0.713 ],
               [ 1.0, 13.0,  1.0   ],
               [ 2.0,  2.0,  0.0   ]])


d1 = pd.DataFrame(a1)

d2 = pd.DataFrame(a2)

d3 = d2.set_index([0,1])[[2]].combine_first(d1.set_index([0,1])[[2]]).reset_index().to_numpy()
d3

Output:

array([[ 0.   ,  2.   ,  0.   ],
       [ 0.   ,  2.   ,  0.   ],
       [ 0.   ,  3.   ,  0.713],
       [ 1.   ,  3.   ,  0.713],
       [ 1.   , 13.   ,  1.   ],
       [ 2.   ,  2.   ,  0.   ],
       [ 2.   ,  5.   , 10.   ]])
user_63615310
Related