'value is not a valid dict' when using pydantic on data loaded from a numpy archive

Drphoton 0
Viewed 49

I've gotten into using pydantic to keep my types in order, but I've been finding that it doesn't play nicely with numpy types. The process of saving objects wraps everything in array of dtype=object and I need to manually undo it everywhere.

I've figured out how to parse this away for strings or lists. For example, for strings, the following seems to work:

from pydantic import BaseModel, validators


class str_type(str):

    @classmethod
    def __get_validators__(cls):
        yield cls.validate
        yield validators.str_validator

    @classmethod
    def validate(cls, value):
        if issubclass(type(value), np.ndarray):
            value = value.item()
        return value


class MyClass(BaseModel):
    my_attribute: str_type

    def save_npz(self, filename):
        """Saves a *.npz file with all attributes"""
        np.savez(filename, **self.dict())

    @classmethod
    def load_npz(cls, filename):
        """Loads a *.npz file and creates an instance of Args"""
        data = np.load(filename, allow_pickle=True)
        data_dict = dict(data)
        return cls(**data_dict)


b = MyClass(my_attribute='hi')

b.save_npz('temp_file.npz')

c = MyClass.load_npz('temp_file.npz')

However, for this trick doesn't seem to work for dicts, and I'm at a loss for why. This is my MWE:

from pydantic import BaseModel, validators


class dict_type(dict):

    @classmethod
    def __get_validators__(cls):
        yield cls.validate
        yield validators.dict_validator

    @classmethod
    def validate(cls, value):
        if issubclass(type(value), np.ndarray):
            value = value.item()
        return value


class MyClass(BaseModel):
    my_attribute: dict_type[str, float]

    def save_npz(self, filename):
        """Saves a *.npz file with all attributes"""
        np.savez(filename, **self.dict())

    @classmethod
    def load_npz(cls, filename):
        """Loads a *.npz file and creates an instance of Args"""
        data = np.load(filename, allow_pickle=True)
        data_dict = dict(data)
        return cls(**data_dict)


b = MyClass(my_attribute={'hi': 3})

b.save_npz('temp_file.npz')

c = MyClass.load_npz('temp_file.npz')

pydantic.error_wrappers.ValidationError: 1 validation error for MyClass my_attribute value is not a valid dict(type=type_error.dict)

EDIT

I ended up using a solution based on @Daniil Fajnberg's suggestions below. I created a standard BaseModel and put a numpy unwrapper within a standard validator that applies to every attribute:

def numpy_unwrap(value):
    """ A common issue when loading data in an *.npz file is that numpy wraps the object in a
    numpy array for safekeeping. For example, instead of saving "True" as type 'bool', it's
    saved as array(True, dtype=object). In most cases, it's easy to unwrap the object from
    this array by just calling the .item() method on the enclosing array, unless it's supposed
    to be a list, in which case you call .tolist().
    """

    if not issubclass(type(value), np.ndarray):
        return value

    try:
        return value.tolist()
    except ValueError:
        return value.item()


class BaseModel_np(BaseModel):

    @validator('*', pre=True, always=True)
    def unwrap_numpy_array(cls, value):
        return numpy_unwrap(value)

    class Config:
        arbitrary_types_allowed = True
pydantic numpy python
1 Answers

To be honest, I don't really see the point in defining your own data type here. You can accomplish what you are trying to do with "vanilla" types and a @validator-decorated method in your model. Here is my suggestion:

from typing import Union

import numpy as np
from pydantic import BaseModel, validator


class MyClass(BaseModel):
    my_attribute: dict[str, float]

    @validator("my_attribute", pre=True)
    def convert_numpy_array(cls, v: Union[dict[str, float], np.ndarray]) -> dict[str, float]:
        if isinstance(v, np.ndarray):
            v = v.item()
        assert isinstance(v, dict)
        return v

    def save_npz(self, filename: str) -> None:
        """Saves a *.npz file with all attributes"""
        np.savez(filename, **self.dict())

    @classmethod
    def load_npz(cls, filename: str) -> "MyClass":
        """Loads a *.npz file and creates an instance of Args"""
        data = np.load(filename, allow_pickle=True)
        return cls.parse_obj(dict(data))


if __name__ == '__main__':
    b = MyClass(my_attribute={'hi': 3})
    b.save_npz('temp_file.npz')
    c = MyClass.load_npz('temp_file.npz')
    print(c)

Output: my_attribute={'hi': 3.0}

Notice that the validator takes care of ensuring a dict instance and does so before other validation thanks to pre=True. If we didn't use that parameter, a ValidationError would be raised because the model's built-in dict-validator would rightly reject the np.ndarray. (The custom validator would never even be called.)

Miscellaneous side notes:

  • I took the liberty of adding some more type annotations since you seem to care about your types.
  • Your check if the type of value is a subclass of np.ndarray is equivalent to simply checking if the value is an instance of np.ndarray. I find the latter more readable.
  • I adjusted your load_npz method to call parse_obj on the dictionary, again because of slightly better readability (in my opinion) than dictionary unpacking.

PS - Custom Base Model:

Since you mentioned you want the solution to be universal, so that you can reuse it anywhere, I suggest defining your own base model with a universal validator. Here is a simple demonstration:

from typing import TypeVar, Union, cast

import numpy as np
from pydantic import BaseModel, validator


T = TypeVar("T")
M = TypeVar("M", bound="MyBaseModel")


class MyBaseModel(BaseModel):
    @validator("*", pre=True)
    def convert_numpy_array(cls, v: Union[T, np.ndarray]) -> T:
        if isinstance(v, np.ndarray):
            v = v.item()
        return cast(T, v)

    def save_npz(self, filename: str) -> None:
        """Saves a *.npz file with all attributes"""
        np.savez(filename, **self.dict())

    @classmethod
    def load_npz(cls: M, filename: str) -> M:
        """Loads a *.npz file and creates an instance of Args"""
        data = np.load(filename, allow_pickle=True)
        return cls.parse_obj(dict(data))


class MyClassA(MyBaseModel):
    some_string: str


class MyClassB(MyClassA):
    a_number: float
    my_dict: dict[str, float]


if __name__ == '__main__':
    b = MyClassB(
        some_string="foo",
        a_number=3.14,
        my_dict={'hi': 3}
    )
    b.save_npz('temp_file.npz')
    c = MyClassB.load_npz('temp_file.npz')
    print(c)

Output: some_string='foo' a_number=3.14 my_dict={'hi': 3.0}

The validator decorator can take the special string "*" instead of the names of specific fields. That way it will be called for every field of the model. The logic inside the validator is pretty much unchanged.

In this example, I put the saving and loading methods into the base model because that seemed sensible, but you can obviously do that as you need it.

That way you can just inherit from MyBaseModel (or from classes that in turn inherit from MyBaseModel) everywhere in your code, and the validator logic will remain intact for every field, as demonstrated by MyClassB in the example.

Of course you may run into problems, if the ndarray.item() method fails for some array. I don't know what your specific requirements are. But you can put in place additional checks in your universal validator, such as checking appropriate dtype of the array or what have you.

The advantage of doing it this way is that inheritance feels much more natural than specifying special types that just introduce additional validation logic. It is less error prone because all you have to do is remember to inherit from your base model. And you can even override validator methods in child models, if you want to.

(The TypeVars and cast function are just type-sugar. Maybe overkill for you.)

Daniil Fajnberg0
Related