Allow `types.GeneratorType` to be used instead of `collections.abc.Generator` as the return type of generator functions

[x42005e1f]

Feature

Currently, mypy only allows collections.abc.Generator and its supertypes as the return type of generator functions, and as a result, it cannot infer the correct return type for awaitable objects that define the __await__() method via types.GeneratorType:

import asyncio

from collections.abc import Generator
from types import GeneratorType
from typing import Never, reveal_type


class A:
    def __await__(self) -> GeneratorType[Never, Never, bool]:
        return True
        yield  # generator definition


class B:
    def __await__(self) -> Generator[Never, Never, bool]:
        return True
        yield  # generator definition


async def main() -> None:
    reveal_type(await A())  # revealed type is "Any"
    reveal_type(await B())  # revealed type is "builtins.bool"

    reveal_type(A().__await__().gi_running)  # revealed type is "builtins.bool"
    reveal_type(B().__await__().gi_running)  # has no attribute "gi_running"


if __name__ == "__main__":
    asyncio.run(main())

example.py:9: error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]
example.py:21: note: Revealed type is "Any"
example.py:22: note: Revealed type is "builtins.bool"
example.py:24: note: Revealed type is "builtins.bool"
example.py:25: error: "Generator[Never, Never, bool]" has no attribute "gi_running"  [attr-defined]
example.py:25: note: Revealed type is "Any"
Found 2 errors in 1 file (checked 1 source file)

I propose allowing the use of types.GeneratorType with subsequent inference of the correct return type.

Pitch

First, pyright already has the expected behavior:

/home/user/workspace/example.py
  /home/user/workspace/example.py:21:17 - information: Type of "await A()" is "bool"
  /home/user/workspace/example.py:22:17 - information: Type of "await B()" is "bool"
  /home/user/workspace/example.py:24:17 - information: Type of "A().__await__().gi_running" is "bool"
  /home/user/workspace/example.py:25:17 - information: Type of "B().__await__().gi_running" is "Unknown"
  /home/user/workspace/example.py:25:33 - error: Cannot access attribute "gi_running" for class "Generator[Never, Never, bool]"
    Attribute "gi_running" is unknown (reportAttributeAccessIssue)
1 error, 0 warnings, 4 informations

Second, this relates to the eternally controversial python/typing#1480. In python/typeshed#10816, false properties related to native objects were removed in favor of the types module. You could say that we can use isinstance/inspect.isgenerator to narrow collections.abc.Generator down to types.GeneratorType, but... it is not that simple:

    x = B().__await__()
    reveal_type(x)  # typing.Generator[Never, Never, builtins.bool]

    if isgenerator(x):
        reveal_type(x)  # types.GeneratorType[Any, Any, Any]

    if isinstance(x, GeneratorType):
        reveal_type(x)  # types.GeneratorType[Any, Any, Any]

I think that allowing the use of types.GeneratorType would make it easier to inspect native generator objects. Especially since collections.abc.Generator can also be used to define custom generator types.

Related: #18635.

[A5rocks]

Technically this wouldn't be completely correct, as I assume Cython doesn't produce GeneratorType. (that's the case for CoroutineType at least).

However, this would still be useful and I think is a good idea nonetheless.

[x42005e1f]

Technically this wouldn't be completely correct, as I assume Cython doesn't produce GeneratorType. (that's the case for CoroutineType at least).

The same applies to other types from the types module. For example, for types.FunctionType:

>>> import inspect
>>> import gevent
>>> gevent.get_hub
<cyfunction get_hub at 0x7f3112f89300>
>>> inspect.isfunction(gevent.get_hub)  # isinstance(gevent.get_hub, types.FunctionType)
False

A function compiled via Cython is not an instance of types.FunctionType. Moreover, although a function compiled via Nuitka is, it does not allow the __code__ attribute to be set at runtime (whereas in typeshed the attribute is defined as settable). However, at the same time, we do not even have any official ways to check that an object is "function-like" (there is even a private helper for this purpose; note that there is also inspect.isroutine(), but it is for a different purpose and does not cover all cases), and the same is true for all other types, including generator iterators.

In light of the above, it seems to me that the isinstance/inspect.is* checks are counterintuitive. They do not return True for Cython objects (unless the types module itself is compiled, which is a very interesting scenario, by the way), but Cython objects have at least some attributes that their uncompiled counterparts have. One could even say that typeshed attempts to define the classes of the types module for only one of the possible runtime behaviors.

>>> gevent.get_hub.__code__
<code object get_hub at 0x7f3113296fb0, file "src/gevent/_hub_local.py", line 79>
>>> gevent.get_hub.__globals__
{'__name__': 'gevent._gevent_c_hub_local', ...}

So maybe possible false positives are not so bad after all?

[x42005e1f]

I just checked for compliance with types.GeneratorType after cythonize -i -3 (the latest Cython):

#!/usr/bin/env python3

from inspect import isgenerator

# copy-paste from types.py
def _g():
    yield 1

_cygen = _g()
_cygen_type = type(_cygen)


def main():
    generator_attrs = (
        "__name__",
        "__qualname__",
        "gi_code",
        "gi_frame",
        "gi_running",
        "gi_suspended",
        "gi_yieldfrom",
    )

    for name in generator_attrs:
        print(_cygen_type, f"has {name!r}:", hasattr(_cygen, name))

    print(_cygen_type, "is generator type:", isgenerator(_cygen))


if __name__ == "__main__":
    main()

<class '_cython_3_2_4.generator'> has '__name__': True
<class '_cython_3_2_4.generator'> has '__qualname__': True
<class '_cython_3_2_4.generator'> has 'gi_code': True
<class '_cython_3_2_4.generator'> has 'gi_frame': True
<class '_cython_3_2_4.generator'> has 'gi_running': True
<class '_cython_3_2_4.generator'> has 'gi_suspended': False
<class '_cython_3_2_4.generator'> has 'gi_yieldfrom': True
<class '_cython_3_2_4.generator'> is generator type: False

coroutines

#!/usr/bin/env python3

from inspect import iscoroutine

# copy-paste from types.py
async def _c(): pass

_cycoro = _c()
_cycoro_type = type(_cycoro)

_cycoro.close()  # prevent `RuntimeWarning`


def main():
    coroutine_attrs = (
        "__name__",
        "__qualname__",
        "cr_code",
        "cr_frame",
        "cr_running",
        "cr_suspended",
        "cr_await",
        "cr_origin",
    )

    for name in coroutine_attrs:
        print(_cycoro_type, f"has {name!r}:", hasattr(_cycoro, name))

    print(_cycoro_type, "is coroutine type:", iscoroutine(_cycoro))


if __name__ == "__main__":
    main()

<class '_cython_3_2_4.coroutine'> has '__name__': True
<class '_cython_3_2_4.coroutine'> has '__qualname__': True
<class '_cython_3_2_4.coroutine'> has 'cr_code': True
<class '_cython_3_2_4.coroutine'> has 'cr_frame': True
<class '_cython_3_2_4.coroutine'> has 'cr_running': True
<class '_cython_3_2_4.coroutine'> has 'cr_suspended': False
<class '_cython_3_2_4.coroutine'> has 'cr_await': True
<class '_cython_3_2_4.coroutine'> has 'cr_origin': False
<class '_cython_3_2_4.coroutine'> is coroutine type: False

async generators (buggy?)

#!/usr/bin/env python3

from inspect import isasyncgen

# copy-paste from types.py
async def _ag():
    yield

_cyasyncgen = _ag()
_cyasyncgen_type = type(_cyasyncgen)


def main():
    asyncgen_attrs = (
        "__name__",
        "__qualname__",
        "ag_code",
        "ag_frame",
        "ag_running",
        "ag_suspended",
        "ag_await",
    )

    for name in asyncgen_attrs:
        print(_cyasyncgen_type, f"has {name!r}:", hasattr(_cyasyncgen, name))

    print(_cyasyncgen_type, "is asyncgen type:", isasyncgen(_cyasyncgen))


if __name__ == "__main__":
    main()

/home/user/workspace/example.c:9811:1: warning: ‘__Pyx_PyAsyncGen_Fini’ defined but not used [-Wunused-function]
 9811 | __Pyx_PyAsyncGen_Fini(void)
      | ^~~~~~~~~~~~~~~~~~~~~
/home/user/workspace/example.c:9230:31: warning: ‘__Pyx__Coroutine_New’ defined but not used [-Wunused-function]
 9230 | static __pyx_CoroutineObject *__Pyx__Coroutine_New(
      |                               ^~~~~~~~~~~~~~~~~~~~
<class '_cython_3_2_4.async_generator'> has '__name__': True
<class '_cython_3_2_4.async_generator'> has '__qualname__': True
<class '_cython_3_2_4.async_generator'> has 'ag_code': False
<class '_cython_3_2_4.async_generator'> has 'ag_frame': False
<class '_cython_3_2_4.async_generator'> has 'ag_running': True
<class '_cython_3_2_4.async_generator'> has 'ag_suspended': False
<class '_cython_3_2_4.async_generator'> has 'ag_await': True
<class '_cython_3_2_4.async_generator'> is asyncgen type: False
sys:1: RuntimeWarning: coroutine '_ag' was never awaited

[x42005e1f]

Looking at it from another angle... it is even more ambiguous.

Les us start with the fact that typing itself is not platform-independent. For example, the following code will not pass type-checking on any platform when both lines are included:

import os

from typing import reveal_type

reveal_type(os.listdrives)  # fails on non-Windows
reveal_type(os.sync)  # fails on non-Unix

But what should be considered the current implementation of Python? Well, obviously, types.pyi (typeshed) is defined for CPython. Moreover, it is not compatible with PyPy:

#!/usr/bin/env python3

import sys

from types import MethodType, MethodWrapperType

if sys.version_info >= (3, 11):  # python/cpython#90633
    from typing import assert_never
else:  # typing-extensions>=4.1.0
    from typing_extensions import assert_never


def main() -> None:
    # funny behavior:
    #   reveal_type(main.__call__)
    #   error: "Callable[[], None]" not callable  [operator]
    #   note: Revealed type is "Any"

    methodwrapper = object().__str__

    if isinstance(methodwrapper, MethodWrapperType):  # always passes
        # reveal_type(methodwrapper.__objclass__)  # builtins.type
        print(hasattr(methodwrapper, "__objclass__"))
        # CPython: True
        # PyPy: False

        if isinstance(methodwrapper, MethodType):
            # fails on PyPy (but it passes mypy & pyright checks)
            assert_never(methodwrapper)

if __name__ == "__main__":
    main()

Cython is not generally considered a separate Python implementation, but as in my previous comment, it changes the behavior of compiled code, making it incompatible with CPython-level execution. Should it even be considered from a typing perspective if the current state of typing is not suitable for anything other than CPython on the current platform?