How to determine the yield / send / return values for any generator in python

user_3130747 0
Viewed 346

Looking at the following: https://docs.python.org/3/library/typing.html#typing.Generator

I'm not sure how one would know what the yield/send/return value is for any given generator?

The yield value seems easy, as I assume one can just do type(next(<generator>)), but the others aren't so clear.

example

For example, I'm not sure how one would typehint the following :

# note - this is not meant to be useful...

import pathlib

def list_of_paths(paths) -> list[pathlib.Path]:
    return list(paths)

path_list = list_of_paths(paths = pathlib.Path('.').glob('*'))

How to know what to use in Generator?


from typing import Generator
import pathlib

def list_of_paths(paths : Generator[pathlib.Path, ? , ? ]) -> list[pathlib.Path]:
    return list(paths)

path_list = list_of_paths(paths = pathlib.Path('.').glob('*'))

edit

Continuing with the example of pathlib.Path.glob, the documentation is https://docs.python.org/3/library/pathlib.html#pathlib.Path.glob

It's note clear here what the send / return types for this generator are.

mypy type-hinting python
2 Answers

Honestly, I rarely ever use Generator. If you're type hinting and only care about the yield value (which is usually the case I find), just use Iterator. A generator is an iterator with extra stuff, so that's a valid hint. In this case you'd just use:

def list_of_paths(paths: Iterator[pathlib.Path]) -> list[pathlib.Path]:
    return list(paths)

To attempt to literally answer the question though, you'd need to dig into the source/documentation of the function and try to reason about what the types are. For glob specifically, if I load this into Pycharm:

import pathlib

p = pathlib.Path()
x = p.glob()

and if I mouse-over x and press ctrl+shift+p (by default), I get a little dialog that says Generator[Path, None, None], which is Pycharm's inference.

Also, if I mouse-over glob and press ctrl+b, I get taken to the source of glob, which is:

def glob(self, pattern):
    sys.audit("pathlib.Path.glob", self, pattern)
    if not pattern:
        raise ValueError("Unacceptable pattern: {!r}".format(pattern))
    drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
    if drv or root:
        raise NotImplementedError("Non-relative patterns are unsupported")
    selector = _make_selector(tuple(pattern_parts), self._flavour)
    for p in selector.select_from(self):
        yield p

And from that, I can tell that it doesn't receive or return any values, so both of those must be None. So basically, if you want to fill in those fields, you'll need to either rely on documentation, a good static analysis tool like Pycharm, or check the source and infer it yourself.

user_30002060

The generator runtime type implements two related but separate concept:

The reason why generator implements both is that a "simple generator" can just be implemented as a "generator coroutine" that replies with its values to empty messages.

As a result, typing.Generator also represents both concepts:

  • Generator[YieldType, SendType, ReturnType] encodes "generator coroutines", and
  • Generator[YieldType, None, None] encodes "simple generators".

So just because one encounters a generator or typing.Generator does not mean there are send and return types.

It is generally viable to assume that a "generator" means a "simple generator" unless explicitly indicated otherwise. The Python glossary: generator only describes the "simple generator" kind, and Python glossary: coroutine refers to async functions – a bare "generator coroutine" is very rare in Python usage.

Given a description like that of pathlib.Path.glob which says "yielding all matching files" and never mentions send or return values, it is safe to assume one deals with a "simple generator".

When in doubt, use MyPy's reveal_type to show the static type of an expression:

$ mypy -c "from pathlib import Path; reveal_type(Path('.').glob('x'))"
<string>:1: note: Revealed type is 'typing.Generator[pathlib.Path*, None, None]'
user_53499160
Related