Calling __new__ on an Any

user_5986907 0
Viewed 792

I'm trying to implement this answer for custom deepcopy, but with type hints, and mypy's not happy with the Any type that I'm using from a third party library. Here's the smallest bit of code I can get to fail

# I'm actually using tensorflow.Module, not Any,
# but afaik that's the same thing. See context below
T = TypeVar("T", bound=Any)

def foo(x: T) -> None:
    cls = type(x)
    cls.__new__(cls)

I see

 error: No overload variant of "__new__" of "type" matches argument type "Type[Any]"
 note: Possible overload variants:
 note:     def __new__(cls, cls: Type[type], o: object) -> type
 note:     def __new__(cls, cls: Type[type], name: str, bases: Tuple[type, ...], namespace: Dict[str, Any]) -> type

It passes if I bound T to something typed, like int, str or a custom class. I'm confused about this, cos neither of these overloads matches the __new__ docs. My knowledge of __new__ is fairly basic.

I'm after either a fix, or if it's a limitation/bug in mypy, an explanation of what that is.

Context

The actual function is

import tensorflow as tf

T = TypeVar("T", bound=tf.Module)  # tf.Module is untyped

def my_copy(x: T, memo: Dict[int, object]) -> T:
    do_something_with_a_tf_module(x)

    cls = type(x)
    new = cls.__new__(cls)
    memo[id(self)] = new

    for name, value in x.__dict__.items():
        setattr(new, name, copy.deepcopy(value, memo))

    return new

curiously, If I instead make this a method

class Mixin(tf.Module):
    def __deepcopy__(self: T, memo: Dict[int, object]) -> T:
        ...  # the same implementation as `my_copy` above

there's no error

type-hinting deep-copy python
1 Answers

The __ new __ suggestions you're getting from mypy are for the type class itself. You can see the constructors match perfectly: https://docs.python.org/3/library/functions.html#type

class type(object)
class type(name, bases, dict)

The complaint by mypy technically makes sense, because you're calling __ new __ from an object returned by type(). If we get the __ class __ of such an object (such as cls), we'll get <class 'type'>:

>>> x = [1,2,3,4]
>>> type(x)
<class 'list'>
>>> type(x).__class__
<class 'type'>

This might be what's tripping up mypy when T is unbounded (i.e. not specified at compile time). If you were inside a class, as you've noticed, and as mentioned in PEP 484 (https://www.python.org/dev/peps/pep-0484/#annotating-instance-and-class-methods), it would be possible for mypy to discern the type as being the class of self, which is unambiguous.

With a standalone function, there are three approaches. One is to silence mypy directly with comment # type:ignore . The second is to grab __ class __ directly from x instead of using type(x), which generally returns the __ class __ anyway (see link above). The third is to use the fact that __ new __ is a class method and call it with x itself.

As long you want to use Any, there's no way to clarify to mypy that type(x) is anything other than Type[T`-1] while maintaining the generic nature (for instance, you could notate the cls = type(x) line with something like # type: List[int], but that would defeat the purpose), and it seems to be resolving the ambiguity with the return type of the type() command.

This coding works for a list (with a silly, element-wise list copy) and keeps me from getting any mypy errors:

from typing import TypeVar, Any, cast
T = TypeVar("T", bound=Any)

def foo(x: T) -> T:
    cls = type(x)
    res = cls.__new__(cls) # type: ignore
    for v in x:
      res.append(v)
    return res

x = [1,2,3,4]
y = foo(x)
y += [5]
print(x)
print(y)

Prints:

[1, 2, 3, 4]
[1, 2, 3, 4, 5]

Alternatively:

def foo(x: T) -> T:
    cls = x.__class__
    res = cls.__new__(cls)
    for v in x:
      res.append(v)
    return res

Third Approach:

from typing import TypeVar, Any
T = TypeVar("T", bound=Any)

def foo(x: T) -> T:
    res = x.__new__(type(x))
    for v in x:
      res.append(v)
    return res

x = [1,2,3,4]
y = foo(x)
y += [5]
print(x)
print(y)
user_151616400
Related