How to pass ForwardRef as args to TypeVar in Python 3.6?

user_10237506 0
Viewed 1039

I'm working on a library that currently supports Python 3.6+, but having a bit of trouble with how forward references are defined in the typing module in Python 3.6. I've setup pyenv on my local Windows machine so that I can switch between different Python versions at ease for local testing, as my system interpreter defaults to Python 3.9.

The use case here essentially is I'm trying to define a TypeVar with the valid forward reference types, which I can then use for type annotation purposes. I've confirmed the following code runs without issue when I'm on 3.7+ and import ForwardRef from the typing module directly, but I'm unable to get it on Python 3.6 since I noticed forward refs can't be used as arguments to TypeVar for some reason. I also tried passing the forward ref type as an argument to Union , but I ran into a similar issue.

Here are the imports and the definition forTypeVar that I'm trying to get to work on both python 3.6.0 as well as more recent versions like 3.6.8 - I did notice I get different errors between minor versions:

from typing import _ForwardRef as PyForwardRef, TypeVar


# Errors on PY 3.6:
#   3.6.2+ -> AttributeError: type object '_ForwardRef' has no attribute '_gorg'
#   3.6.2 or earlier -> AssertionError: assert isinstance(a, GenericMeta)
FREF = TypeVar('FREF', str, PyForwardRef)

Here is a sample usage I've been able to test out, which appears to type check as expected for Python 3.7+:

class MyClass: ...


def my_func(typ: FREF):
    pass


# Type checks
my_func('testing')
my_func(PyForwardRef('MyClass'))

# Does not type check
my_func(23)
my_func(MyClass)

What I've Done So Far

Here's my current workaround I'm using to support Python 3.6. This isn't pretty but it seems to at least get the code to run without any errors. However this does not appear to type check as expected though - at least not in Pycharm.

import typing

# This is needed to avoid an`AttributeError` when using PyForwardRef
# as an argument to `TypeVar`, as we do below.
if hasattr(typing, '_gorg'):  # Python 3.6.2 or lower
    _gorg = typing._gorg
    typing._gorg = lambda a: None if a is PyForwardRef else _gorg(a)
else:  # Python 3.6.3+
    PyForwardRef._gorg = None

Wondering if I'm on the right track, or if there's a simpler solution I can use to support ForwardRef types as arguments to TypeVar or Union in Python 3.6.

python-typing type-hinting forward-declaration python-3.x python
1 Answers

To state the obvious, the issue here appears to be due to several changes in the typing module between Python 3.6 and Python 3.7.

In both Python 3.6 and Python 3.7:

  • All constraints on a TypeVar are checked using the typing._type_check function (links are to the 3.6 branch of the source code on GitHub) before the TypeVar is allowed to be instantiated.


    TypeVar.__init__ looks something like this in the 3.6 branch:

    class TypeVar(_TypingBase, _root=True):

    # <-- several lines skipped -->

    def __init__(self, name, *constraints, bound=None,
             covariant=False, contravariant=False):

        # <-- several lines skipped -->

        if constraints and bound is not None:
            raise TypeError("Constraints cannot be combined with bound=...")
        if constraints and len(constraints) == 1:
            raise TypeError("A single constraint is not allowed")
        msg = "TypeVar(name, constraint, ...): constraints must be types."
        self.__constraints__ = tuple(_type_check(t, msg) for t in constraints)

        # etc.

In Python 3.6:

  • There was a class called _ForwardRef. This class was given a name with a leading underscore to warn users that it was an implementation detail of the module, and that therefore the API of the class could change unexpectedly between Python versions.
  • It appears that typing._type_check did not account for the possibility that _ForwardRef might be passed to it, hence the strange AttributeError: type object '_ForwardRef' has no attribute '_gorg' error message. I assume that this possibility was not accounted for because it was assumed that users would know not to use classes marked as implementation details.

In Python 3.7:

  • _ForwardRef has been replaced with a ForwardRef class: this class is no longer an implementation detail; it is now part of the module's public API.

  • typing._type_check now explicitly accounts for the possibility that ForwardRef might be passed to it:

    def _type_check(arg, msg, is_argument=True):
    """Check that the argument is a type, and return it (internal helper).
    As a special case, accept None and return type(None) instead. Also wrap strings
    into ForwardRef instances. Consider several corner cases, for example plain
    special forms like Union are not valid, while Union[int, str] is OK, etc.
    The msg argument is a human-readable error message, e.g::
        "Union[arg, ...]: arg should be a type."
    We append the repr() of the actual value (truncated to 100 chars).
    """

    # <-- several lines skipped -->

    if isinstance(arg, (type, TypeVar, ForwardRef)):
        return arg

    # etc.

Solutions

I'm tempted to argue that it's not really worth the effort to support Python 3.6 at this point, given that Python 3.6 is kind of old now, and will be officially unsupported from December 2021. However, if you do want to continue to support Python 3.6, a slightly cleaner solution might be to monkey-patch typing._type_check rather than monkey-patching _ForwardRef. (By "cleaner" I mean "comes closer to tackling the root of the problem, rather than a symptom of the problem" — it's obviously less concise than your existing solution.)

import sys 
from typing import TypeVar

if sys.version_info < (3, 7):
    import typing
    from typing import _ForwardRef as PyForwardRef
    from functools import wraps

    _old_type_check = typing._type_check

    @wraps(_old_type_check)
    def _new_type_check(arg, message):
        if arg is PyForwardRef:
            return arg
        return _old_type_check(arg, message)

    typing._type_check = _new_type_check
    # ensure the global namespace is the same for users
    # regardless of the version of Python they're using
    del _old_type_check, _new_type_check, typing, wraps
else:
    from typing import ForwardRef as PyForwardRef

However, while this kind of thing works fine as a runtime solution, I have honestly no idea whether there is a way to make type-checkers happy with this kind of monkey-patching. Pycharm, MyPy and the like certainly won't be expecting you to do something like this, and probably have their support for TypeVars hardcoded for each version of Python.

user_139900160
Related