Perform generic functions on a specific typed REST endpoint keeping the types

Question

I am making a typed REST library, where all the endpoints have specific classes and have their methods set on the object. Say we have a list of strings returned by the endpoint A, it would have the MVCE class A below. I add methods that all endpoints need to function in the Base class so that the endpoints include as little boilerplate as possible.

However there are some functions that I need to perform on all 'list' endpoints, such as A and B below, but not C. This common function is get_all, so that we get all the objects from the list.

The problem is I have the code working, however PyCharm and mypy doesn't know the type of a or b, and says the type is List[T], this makes sense as I haven't specified what T is.

How can I make a have the type List[str], and b have the type List[int]?

_mock_a = list('abcdefghijklmnopqrstuvwxyz')
_mock_b = [int(i) for i in '12345678901234567890123456']

from typing import TypeVar, Callable, List

T = TypeVar('T')


class Base:
    def pipe(self, fn: Callable[['Base'], List[T]]) -> List[T]:
        return fn(self)


class A(Base):
    def get(self, index=0, count=5) -> List[str]:
        return _mock_a[index:index+count]

    def count(self) -> int:
        return len(_mock_a)


class B(Base):
    def get(self, index=0, count=5) -> List[int]:
        return _mock_b[index:index+count]

    def count(self) -> int:
        return len(_mock_b)


class C(Base):
    def other(self) -> None:
        pass


def get_all(base: Base) -> List[T]:
    step = 5
    return [
        item
        for start in range(0, base.count(), step)
        for item in base.get(start, step)
    ]


# Has type List[T], but I want it to have List[str]
a = A().pipe(get_all)
print(a)
# Has type List[T], but I want it to have List[int]
b = B().pipe(get_all)
print(b)

I have tried the following to fix this, but neither worked.

class Method(Generic[T]):
    @staticmethod
    def get_all(base: Base) -> List[T]:
        step = 5
        return [
            item
            for start in range(0, base.count(), step)
            for item in base.get(start, step)
        ]


a = A().pipe(Method[str].get_all)
print(a)

class Base:
    def pipe(self, t: Type[T], fn: Callable[['Base'], T]) -> T:
        return fn(self)


a = A().pipe(List[str], get_all)
print(a)

I have found a way to get the second working, which works like typing.cast:

class Base:
    def pipe(self, fn: Callable[['GetableEndpoint[T]'], List[T]], t: Type[T]=T) -> List[T]:
        return fn(cast(GetableEndpoint[T], self))


class GetableEndpoint(Generic[T], Base, metaclass=abc.ABCMeta):
    @classmethod
    def __subclasshook__(cls, C):
        if cls is GetableEndpoint:
            if any('get' in B.__dict__ for B in C.__mro__) and any('count' in B.__dict__ for B in C.__mro__):
                return True
        return NotImplemented

    @abc.abstractmethod
    def get(self, index=0, count=5) -> List[T]:
        raise NotImplementedError()

    @abc.abstractmethod
    def count(self) -> int:
        raise NotImplementedError()


def get_all(base: GetableEndpoint[T]) -> List[T]:
    step = 5
    return [
        item
        for start in range(0, base.count(), step)
        for item in base.get(start, step)
    ]


a = A().pipe(get_all, str)

user_2044866 · Answer

The question Python type annotation for custom duck type is like this question, and included a link to Protocols (a.k.a. structural subtyping). This issue created PEP 544, which has an implementation in typing_extensions.

This means that to fix the above we can change GetableEndpoint to be a Protocol.

from typing import TypeVar, List
from typing_extensions import Protocol
import abc

T = TypeVar('T')


class GetableEndpoint(Protocol[T]):
    @abc.abstractmethod
    def get(self, index=0, count=5) -> List[T]:
        pass

    @abc.abstractmethod
    def count(self) -> int:
        pass

This allows the fully typed usage in both PyCharm and Mypy of:

class A(Base):
    def get(self, index=0, count=5) -> List[str]:
        return _mock_a[index:index+count]

    def count(self) -> int:
        return len(_mock_a)


def get_all(base: GetableEndpoint[T]) -> List[T]:
    step = 5
    return [
        item
        for start in range(0, base.count(), step)
        for item in base.get(start, step)
    ]


a = get_all(A())
print(a)

I can't get pipe to work, however this now has fully working types which I think is more important.

Perform generic functions on a specific typed REST endpoint keeping the types

1 Answers