Calculate distance between arrays that contain NaN

user_11225028 0
Viewed 537

consider array1 and array2, with:

array1 = [a1 a2 NaN ... an]
array2 = [[NaN b2 b3 ... bn],
          [b21 NaN b23 ... b2n],
          ...]

Both arrays are numpy-arrays. There is an easy way to compute the Euclidean distance between array1and each row of array2: 

EuclideanDistance = np.sqrt(((array1 - array2)**2).sum(axis=1))

What messes up this computation are the NaN values. Of course, I could easily replace NaN with some number. But instead, I want to do the following:

When I compare array1 with row_x of array2, I count the columns in which one of the arrays has NaN and the other doesn't. Let's assume the count is 3. I will then delete these columns from both arrays and compute the Euclidean distance between the two. In the end, I add a minus_value * count to the calculated distance.

Now, I cannot think of a fast and efficient way to do this. Can somebody help me?

Here are a few of my ideas: 

minus = 1000
dist = np.zeros(shape=(array1.shape[0])) # this array will store the distance of array1 to each row of array2
array1 = np.repeat(array1, array2.shape[0], axis=0) # now array1 has the same dimensions as array2
for i in range(0, array1.shape[0]):
    boolarray = np.logical_or(np.isnan(array1[i]), np.isnan(array2[i]))
    count = boolarray.sum()
    deleteIdxs = boolarray.nonzero() # this should give the indices where boolarray is True
    dist[i] = np.sqrt(((np.delete(array1[i], deleteIdxs, axis=0) - np.delete(array2[i], deleteIdxs, axis=0))**2).sum(axis=0))
    dist[i] = dist[i] + count*minus

These lines look more than ugly to me, however. Also, I keep getting an index error: Apparently deleteIdxs contains an index that is out of range for array1. Don't know how this can even be.

numpy python
3 Answers

You can find all the indices with where the value is nan using:

indices_1 = np.isnan(array1)
indices_2 = np.isnan(array2)

Which you can combine to:

indices_total = indices_1 + indices_2

And you can keep all the not nan values using:

array_1_not_nan = array1[~indices_total]
array_2_not_nan = array2[~indices_total]
user_67664570

I would write a function to handle the distance calculation. I am sure there is a faster and more efficient way to write this (list comprehensions, aggregations, etc.), but readability counts, right? :) 

import numpy as np
def calculate_distance(fixed_arr, var_arr, penalty):
    s_sum = 0.0
    counter = 0
    for num_1, num_2 in zip(fixed_arr, var_arr):
        if np.isnan(num_1) or np.isnan(num_2):
            counter += 1
        else:
            s_sum += (num_1 - num_2) ** 2
    return np.sqrt(s_sum) + penalty * counter, counter


array1 = np.array([1, 2, 3, np.NaN, 5, 6])
array2 = np.array(
    [
        [3, 4, 9, 3, 4, 8],
        [3, 4, np.NaN, 3, 4, 8],
        [np.NaN, 9, np.NaN, 3, 4, 8],
        [np.NaN, np.NaN, np.NaN, np.NaN, np.NaN, np.NaN],
    ]
)
dist = np.zeros(len(array2))


minus = 10
for index, arr in enumerate(array2):
    dist[index], _ = calculate_distance(array1, arr, minus)

print(dist)

You have to think about the value for the minus variable very carefully. Is adding a random value really useful?

As @Nathan suggested, a more resource efficient can easily be implemented. 

fixed_arr = array1
penalty = minus
dist = [
    (
        lambda indices=(np.isnan(fixed_arr) + np.isnan(var_arr)): np.linalg.norm(
            fixed_arr[~indices] - var_arr[~indices]
        )
        + (indices == True).sum() * penalty
    )()
    for var_arr in array2
]
print(dist)

However I would only try to implement something like this if I absolutely needed to (if it's the bottleneck). For all other times I would be happy to sacrifice some resources in order to gain some readability and extensibility.

user_134258720

You can filter out the columns containing nan with:

mask1 = np.isnan(arr1)
mask2 = np.isnan(arr2).any(0)

mask = ~(mask1 | mask2)

# the two filtered arrays
arr1[mask], arr2[mask]
user_42384080
Related