How to resolve numpy ValueError after changing order?

user_3150024 0
Viewed 235

I have created a numpy array (consider input data) and would like to change the order (output data after some numerical operations). While using converted array, I am getting below error and found the root cause. Please find the details below and using numpy version 1.19.1

import numpy as np
a = np.array([1,2,3])
b = np.array([(1.5,2,3), (4,5,6)], dtype = [('col1','<f8'),('col2','<i4'),('col3','<i4')])
names_list=['col2','col1','col3']
c=b[names_list]
 
print(c.dtype.descr)

But I am getting below error after changing the order of the fields.

lib\site-packages\numpy\core_internal.py", line 114, in _array_descr raise ValueError( ValueError: dtype.descr is not defined for types with overlapping or out-of-order fields

I have done some analysis and found that "offsets" value causing this problem. So i displayed the offsets after order change as below,

{'names':['col2','col1','col3'], 'formats':['<i4','<f8','<i4'], 'offsets':[8,0,12], 'itemsize':16}

and then opened numpy library code in lib\site-packages\numpy\core_internal.py,

offset = 0
for field in ordered_fields:
    if field[1] > offset:
        num = field[1] - offset
        result.append(('', '|V%d' % num))
        offset += num
    elif field[1] < offset:
        raise ValueError(
            "dtype.descr is not defined for types with overlapping or "
            "out-of-order fields")

Thanks in advance !

numpy python
1 Answers
In [73]: b = np.array([(1.5,2,3), (4,5,6)], dtype = [('col1','<f8'),('col2','<i4'),('col3','<i4')])  
In [74]: b                                                                                           
Out[74]: 
array([(1.5, 2, 3), (4. , 5, 6)],
      dtype=[('col1', '<f8'), ('col2', '<i4'), ('col3', '<i4')])

In recent versions, a multifield index of a structured array returns a view; one which includes the offset as part of the dtype:

In [75]: names_list=['col2','col1','col3']                                                           
In [76]: c = b[names_list]                                                                           
In [77]: c                                                                                           
Out[77]: 
array([(2, 1.5, 3), (5, 4. , 6)],
      dtype={'names':['col2','col1','col3'], 'formats':['<i4','<f8','<i4'], 'offsets':[8,0,12], 'itemsize':16})
In [78]: c.dtype                                                                                     
Out[78]: dtype({'names':['col2','col1','col3'], 'formats':['<i4','<f8','<i4'], 'offsets':[8,0,12], 'itemsize':16})

c is a view, accessing the same data as b, but presenting the fields in the new order (and using offset to do that). As your error says, the fields are out-of-order.

They have provided a repack_fields function if you want an a copy with fields in the new order (without offsets):

In [79]: import numpy.lib.recfunctions as rf                                                         
In [80]:                                                                                             
In [80]: d = rf.repack_fields(c)                                                                     
In [81]: d                                                                                           
Out[81]: 
array([(2, 1.5, 3), (5, 4. , 6)],
      dtype=[('col2', '<i4'), ('col1', '<f8'), ('col3', '<i4')])
In [82]: d.dtype                                                                                     
Out[82]: dtype([('col2', '<i4'), ('col1', '<f8'), ('col3', '<i4')])
In [83]: d.dtype.descr                                                                               
Out[83]: [('col2', '<i4'), ('col1', '<f8'), ('col3', '<i4')]

Read repack_fields docs for more details. https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields

Your question makes more sense with with savez context:

In [106]: np.savez('struct.npz', b=b, d=d)                                                           
In [107]: np.savez('struct.npz', c=c)                                                                
...
ValueError: dtype.descr is not defined for types with overlapping or out-of-order fields

So if you want to save the array with different field order, you need to use repack to get a "clean" copy.

user_9019250
Related