`in` defined for generator

Viewed 235

Why is in operator defined for generators?

>>> def foo():
...     yield 42
... 
>>> 
>>> f = foo()
>>> 10 in f
False

What are the possible use cases?

I know that range(...) objects have a __contains__ function defined so that we can do stuff like this:

>>> r = range(10)
>>> 4 in r
True
>>> r.__contains__
<method-wrapper '__contains__' of range object at 0x7f82bd51cc00>

But f above doesn't have __contains__ method.

2 Answers

"What are the possible use cases?" To check if the generator will produce some value.

Dunder methods serve as hooks for the particular syntax they are associated with. __contains__ isn't some kind of one-to-one mapping to x in y. The language ultimately defines the semantics of these operators.

From the documentation of membership testing, we see there are several ways for x in y to be evaluated, depending on various properties of the objects involved. I've highlted the relevant one for generator objects, which do not define a __contains__ but are iterable, i.e., they define an __iter__ method:

The operators in and not in test for membership. x in s evaluates to True if x is a member of s, and False otherwise. x not in s returns the negation of x in s. All built-in sequences and set types support this as well as dictionary, for which in tests whether the dictionary has a given key. For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression x in y is equivalent to any(x is e or x == e for e in y).

For the string and bytes types, x in y is True if and only if x is a substring of y. An equivalent test is y.find(x) != -1. Empty strings are always considered to be a substring of any other string, so "" in "abc" will return True.

For user-defined classes which define the __contains__() method, x in y returns True if y.__contains__(x) returns a true value, and False otherwise.

For user-defined classes which do not define contains() but do define __iter__(), x in y is True if some value z, for which the expression x is z or x == z is true, is produced while iterating over y. If an exception is raised during the iteration, it is as if in raised that exception.

Lastly, the old-style iteration protocol is tried: if a class defines __getitem__(), x in y is True if and only if there is a non-negative integer index i such that x is y[i] or x == y[i], and no lower integer index raises the IndexError exception. (If any other exception is raised, it is as if in raised that exception).

The operator not in is defined to have the inverse truth value of in.

To summarize, x in y will be defined for objects that:

  1. Are strings or bytes, and it is defined as a substring relationship.
  2. types that define __contains__
  3. types that are iterators, i.e. that define __iter__
  4. the old-style iteration protocol (relies on __getitem__)

Generators fall into 3.

A broader point, you really shouldn't use the dunder methods directly, unless you really understand what they are doing. Even then, it may be best to aviod it.

It usually isn't worth trying to be credible or succinct by using something to the effect of:

x.__lt__(y)

Instead of:

x < y

You should at least understand, that this might happen:

>>> (1).__lt__(3.)
NotImplemented
>>>

And if you are just naively doing stuff like filter((1).__lt__, iterable) then you've probably got a bug.

Generators are iterable, and so have an .__iter__ method, which can be used to check membership

How this behaves is described in the Expressions docs on Membership test operations

For user-defined classes which do not define __contains__() but do define __iter__(), x in y is True if some value z, for which the expression x is z or x == z is true, is produced while iterating over y. If an exception is raised during the iteration, it is as if in raised that exception.

Emphasis mine!

This pops the whole generator, and your example with 42 simply doesn't include the tested value of 10

>>> def foo():
...     yield 5
...     yield 10
...
>>> f = foo()
>>> 10 in f
True
>>> 10 in f
False
Related