PEP 585 + PEP 604 support about typing_extensions

[mikeshardmind]

Guess I'll followup here from the original discussion in discord: Personally, I think this would be better suited as typing_extensions.get_type_hints_ignore_unknown_generics, and be a function that could be used as desired that did this rather than an approach that uses monkeypatching of the standard library, which has global effects that not every library will be aware of.

[bswck]

rather than an approach that uses monkeypatching of the standard library, which has global effects that not every library will be aware of.

The suggested backport would introduce no monkeypatching to the standard library, as I've already mentioned above

• here:

  Incorporating the backports would be ideal, because no monkeypatching would have to occur (__modern_types__ reassigns typing.ForwardRef._evaluate with a custom implementation).

• and here:

  My idea would be to export a subclass of ForwardRef and then pretty much the same stuff as in my aforementioned implementation.

The problem of "global effects" you are mentioning is out of scope of the issue.

After incorporating the backport, typing_extensions.ForwardRef would need to be used instead of typing.ForwardRef and PEP 585 and PEP 604 would be supported without any monkeypatching. A clear technical explanation is available in the current implementation README.

[mikeshardmind]

The suggested backport would introduce no monkeypatching to the standard library, as I've already mentioned above
[...]
My idea would be to make a subclass of ForwardRef and then pretty much the same stuff as in my aforementioned implementation.

As already discussed in discord, the primary source of ForwardRef instances is not from direct user instantiation of them, simply providing a subclass with different ._evaluate is insufficient to allow this in forward references, there has to be a mechanism to handle annotations with this. I'm sorry I missed that you're no longer monkey patching at all when initially going to respond, but that only raises further questions about providing a subclass and not an alternative functional method for evaluating/getting type hints with support for future syntax in forward references.

[JelleZijlstra]

I may have to look into this in more detail, but I'm not enthusiastic. @mikeshardmind is right that direct instantiation of ForwardRef is not an important use case. This approach also won't fix PEP 585/604 usage outside of annotations, e.g. in TypeVar bounds, casts, type aliases.

[bswck]

the primary source of ForwardRef instances is not from direct user instantiation of them, simply providing a subclass with different ._evaluate is insufficient to allow this in forward references, there has to be a mechanism to handle annotations with this

@mikeshardmind is right that direct instantiation of ForwardRef is not an important use case.

Yes, this is true. OTOH, patching

typing_extensions/src/typing_extensions.py

Lines 1071 to 1111 in d6dc4f1

	def get_type_hints(obj, globalns=None, localns=None, include_extras=False):
	"""Return type hints for an object.
	This is often the same as obj.__annotations__, but it handles
	forward references encoded as string literals, adds Optional[t] if a
	default value equal to None is set and recursively replaces all
	'Annotated[T, ...]', 'Required[T]' or 'NotRequired[T]' with 'T'
	(unless 'include_extras=True').
	The argument may be a module, class, method, or function. The annotations
	are returned as a dictionary. For classes, annotations include also
	inherited members.
	TypeError is raised if the argument is not of a type that can contain
	annotations, and an empty dictionary is returned if no annotations are
	present.
	BEWARE -- the behavior of globalns and localns is counterintuitive
	(unless you are familiar with how eval() and exec() work). The
	search order is locals first, then globals.
	- If no dict arguments are passed, an attempt is made to use the
	globals from obj (or the respective module's globals for classes),
	and these are also used as the locals. If the object does not appear
	to have globals, an empty dictionary is used.
	- If one dict argument is passed, it is used for both globals and
	locals.
	- If two dict arguments are passed, they specify globals and
	locals, respectively.
	"""
	if hasattr(typing, "Annotated"): # 3.9+
	hint = typing.get_type_hints(
	obj, globalns=globalns, localns=localns, include_extras=True
	)
	else: # 3.8
	hint = typing.get_type_hints(obj, globalns=globalns, localns=localns)
	if include_extras:
	return hint
	return {k: _strip_extras(t) for k, t in hint.items()}

to use typing_extensions-side ForwardRefs does not seem to be uneasy.

The cost of migrating from typing.get_type_hints to typing_extensions.get_type_hints seems to be close to zero, and can enable a lot of simplification.

This approach also won't fix PEP 585/604 usage outside of annotations, e.g. in TypeVar bounds, casts, type aliases.

At the end of the day, there is no need to fix it, because quoting or TYPE_CHECKING-blocks can be used and everything will work just fine.

• For TypeVar bounds:

  Foo = TypeVar("Foo", bound="Callable[..., Bar]")

  Inside a TYPE_CHECKING block, from collections.abc import Callable can be used and bound= argument does not have to be quoted.
• For casts:

  foo = cast("Foo", foo)

  Casting doesn't happen often in TYPE_CHECKING blocks, so quoting is necessary.
• For type aliases:

  from typing_extensions import TypeAlias
  FooCallable: TypeAlias = "Callable[..., Foo]"

  Here—similarly, inside a TYPE_CHECKING block, from collections.abc import Callable can be used and the type alias does not have to be quoted.

TYPE_CHECKING blocks in pairs with from __future__ import annotations are quite often in the modern codebases, and AFAIK the latter is planned to be a default behavior in the future. That means there is no need to focus on solving evaluating PEP 585/604 expressions directly at runtime, due to the possibility of using the techniques listed above.

The problem is when you try to evaluate the new-fashion (3.10+) type hints which for example takes place in pydantic. This is the last puzzle to allow 3.8+ codebases to fully introduce PEP 585/604.

typing_extensions's goal is to provide backports to access typing with features from future Python versions, hence this issue.

[bswck]

For instance, take a look at https://fastapi.tiangolo.com/advanced/additional-status-codes/#additional-status-codes_1.
Patching typing_extensions to work with the new ForwardRefs would really simplify this Union[X, Y] vs X | Y discrepancy and make it easier to learn for the new Python programmers that X | Y is the way to go.
Even apart from this, it can save the extra time spent on importing deprecated things from typing.

I see an insignificant cost and a huge benefit in supporting PEP 585 and PEP 604 as this issue suggests.

[bswck]

Observation 1: the current implementation does not fully implement PEP 604, because cases like non-generic int | None are not handled (int is not replaced by any object in the evaluation namespaces).

Observation 2: the current implementation will have problems with dotted names, such as in cases like the following.

This will work:

from __future__ import annotations

from collections.abc import Mapping

# In this snippet, I am assuming the current implementation is part of `typing_extensions`.
from typing_extensions import get_type_hints

class Foo:
    bar: Mapping[str, int]

get_type_hints(Foo, globals(), locals())

but this will not:

from __future__ import annotations

import collections.abc

from typing_extensions import get_type_hints

class Foo:
    bar: collections.abc.Mapping[str, int]

get_type_hints(Foo, globals(), locals())

If you are about to review the code any soon, please keep that in mind. I am working on that.
Fixed in v2.0.3.

[DanCardin]

For reference, (and for unrelated reasons,) I have a PR that duplicates ForwardRef into typing_extensions using the implementation as of 3.11. Due to it being Final, I found it necessary to copy the whole implementation rather than subclass it.

In my experience, it was not trivial to extricate. Although the fact that doing so revealed several ways in which the <3.11 featureset was quite different, to the point that it seemed like it was a valuable exercise if it meant that typing_extensions would be able to backport the current-version behavior of get_type_hints entirely.

[bswck]

Closing this for now. I'm working on https://github.com/alexmojaki/eval_type_backport with @alexmojaki. Feel free to reopen this if you would like me to work on this in typing extensions.

[alexmojaki]

@DanCardin I just came across DanCardin/cappa#100. I see you added your own version of get_type_hints that's very similar to (~ copied from?) the one in pydantic, i.e. it uses eval_type_backport and handles compatibility for ForwardRefs. Is it fair to say that if this version of get_type_hints existed in typing_extensions you could just use that directly and it would have saved you some trouble?

[DanCardin]

very much so! the call sites of eval_type_backport in get_type_hints are too deeply embedded to avoid re-stating the whole implementation afaict