Boolean mask of None vs NaN in array of arrays

user_7998725 0
Viewed 638

I want to differentiate the None's from NaN's in a NumPy array with mixed data types. Specifically, I want to work with arrays where some elements are arrays. For example, suppose I have the following array with mixed data types, but without array elements:

arr = np.array([[1,    None,      3],
                [4,       5, np.nan],
                [7, 'eight',     9]])

I can check what values are None with the following:

>>> arr == None

array([[False,  True, False],
       [False, False, False],
       [False, False, False]])

This gets the result that I want: a Boolean mask with information about the individual elements, saying which elements are None. However, the snippet above doesn't work when any element in the array is another array. For example, consider this other array:

arr_elem = np.array([3,3,3])
arr = np.array([[1,    None,      arr_elem],
                [4,       5,        np.nan],
                [7, 'eight',             9]])

Now, instead of a Boolean mask, I obtain a single Boolean value:

>>> arr == None

False

What is a vectorized way to obtain a Boolean mask that says which elements are None and not NaN in an array of arrays?

numpy-ndarray multidimensional-array boolean numpy python
1 Answers

You have an object dtype array. numpy arrays are best when the dtype is numeric - both in terms of speed and extent of operations.

On top of that there are issues with testing for None, np.nan and equality tests on arrays.

is None is the best test for None. np.isnan is best test for nan, since nothing equals nan, not even itself. And equality tests on arrays can produce boolean arrays and ambiguity errors.

In [58]: arr = np.array([[1,    None,      3],
    ...:                 [4,       5, np.nan],
    ...:                 [7, 'eight',     9]])
In [59]: arr
Out[59]: 
array([[1, None, 3],
       [4, 5, nan],
       [7, 'eight', 9]], dtype=object)
In [60]: arr == None
Out[60]: 
array([[False,  True, False],
       [False, False, False],
       [False, False, False]])

ok, the None test is ok, but not nan:

In [61]: arr == np.nan
Out[61]: 
array([[False, False, False],
       [False, False, False],
       [False, False, False]])
In [62]: np.isnan(arr)
Traceback (most recent call last):
  File "<ipython-input-62-25d2f1dae78d>", line 1, in <module>
    np.isnan(arr)
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

isnan works with float dtype arrays, not object dtype.

Making an array with array elements produces a warning. That's because making an array from arrays of mixed length often is a mistake.

In [63]: arr_elem = np.array([3,3,3])
    ...: arr1 = np.array([[1,    None,      arr_elem],
    ...:                 [4,       5,        np.nan],
    ...:                 [7, 'eight',             9]])
<ipython-input-63-6d36e87ab1a9>:2: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  arr1 = np.array([[1,    None,      arr_elem],
In [64]: arr1
Out[64]: 
array([[1, None, array([3, 3, 3])],
       [4, 5, nan],
       [7, 'eight', 9]], dtype=object)

This is an object dtype array as before, except that an element is an array.

In [65]: arr1 == None
<ipython-input-65-2a40a08151e4>:1: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
  arr1 == None
Out[65]: False

It has to do the == test on each element. But that test applied to the array element produces a boolean array (same as In[60])

In [66]: arr_elem==None
Out[66]: array([False, False, False])
user_9019250
Related