PEP 718: Specify binding, parametrisation and overload interactions (#4649)

* hopefully final round of changes

* I think fix the build issues?
Sorry I can't test I can't get make installed

* Fix title too short

* fix some grammar typos and build

* Update peps/pep-0718.rst

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

* final round of changes

* Split long sentence in two for style

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

* Update pep-0718.rst

Adresed @gvanrossum comments

* Update pep-0718.rst

Small change

* fix line wrap

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more line wrap

---------

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
Co-authored-by: Pablo <48098178+PabloRuizCuevas@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: 6 people

peps/pep-0718.rst

@@ -1,6 +1,6 @@
 PEP: 718
 Title: Subscriptable functions
-Author: James Hilton-Balfe <gobot1234yt@gmail.com>
+Author: James Hilton-Balfe <gobot1234yt@gmail.com>, Pablo Ruiz Cuevas <pablo.r.c@live.com>
 Sponsor: Guido van Rossum <guido@python.org>
 Discussions-To: https://discuss.python.org/t/28457/
 Status: Draft
@@ -17,68 +17,132 @@ This PEP proposes making function objects subscriptable for typing purposes. Doi
 gives developers explicit control over the types produced by the type checker where
 bi-directional inference (which allows for the types of parameters of anonymous
 functions to be inferred) and other methods than specialisation are insufficient. It
-also brings functions in line with regular classes in their ability to be
-subscriptable.
+also makes functions consistent with regular classes in their ability to be
+subscripted.
 
 Motivation
 ----------
 
-Unknown Types
-^^^^^^^^^^^^^
+Currently, classes allow passing type annotations for generic containers. This
+is especially useful in common constructors such as ``list``\, ``tuple`` and ``dict``
+etc.
 
-Currently, it is not possible to infer the type parameters to generic functions in
-certain situations:
+.. code-block:: python
+
+   my_integer_list = list[int]()
+   reveal_type(my_integer_list)  # type is list[int]
+
+At runtime ``list[int]`` returns a ``GenericAlias`` that can be later called, returning
+an empty list.
+
+Another example of this is creating a specialised ``dict`` type for a section of our
+code where we want to ensure that keys are ``str`` and values are ``int``:
 
 .. code-block:: python
 
-   def make_list[T](*args: T) -> list[T]: ...
-   reveal_type(make_list())  # type checker cannot infer a meaningful type for T
+   NameNumberDict = dict[str, int]
 
-Making instances of ``FunctionType`` subscriptable would allow for this constructor to
-be typed:
+   NameNumberDict(
+      one=1,
+      two=2,
+      three="3"  # Invalid: Literal["3"] is not of type int
+   )
+
+In spite of the utility of this syntax, when trying to use it with a function, an error
+is raised, as functions are not subscriptable.
 
 .. code-block:: python
 
-   reveal_type(make_list[int]())  # type is list[int]
+   def my_list[T](arr: Iterable[T]) -> list[T]:
+      # do something...
+      return list(arr)
+
+   my_integer_list = my_list[int]()  # TypeError: 'function' object is not subscriptable
 
-Currently you have to use an assignment to provide a precise type:
+There are a few workarounds:
+
+1. Making a callable class:
 
 .. code-block:: python
 
-   x: list[int] = make_list()
-   reveal_type(x)  # type is list[int]
+   class my_list[T]:
+        def __call__(self, arr: Iterable[T]) -> list[T]:
+            # do something...
+            return list(arr)
 
-but this code is unnecessarily verbose taking up multiple lines for a simple function
-call.
+   my_string_list = my_list[str]([])
 
-Similarly, ``T`` in this example cannot currently be meaningfully inferred, so ``x`` is
-untyped without an extra assignment:
+2. Using :pep:`747`\'s TypeForm, with an extra unused argument:
 
 .. code-block:: python
 
-   def factory[T](func: Callable[[T], Any]) -> Foo[T]: ...
+   from typing import TypeForm
 
-   reveal_type(factory(lambda x: "Hello World" * x))
+   def my_list(*arr: Iterable[T], typ: TypeForm[T]) -> list[T]:
+      # do something...
+      return list(arr)
 
-If function objects were subscriptable, however, a more specific type could be given:
+   my_string_list = my_list([], str)
+
+As we can see this solution increases the complexity with an extra argument.
+Additionally it requires the user to understand a new concept ``TypeForm``.
+
+3. Annotating the assignment:
 
 .. code-block:: python
 
-   reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]
+   my_integer_list: list[int] = my_list()
+
+This solution isn't optimal as the return type is repeated, is more verbose and would
+require the type updating in multiple places if the return type changes. Additionally,
+it adds unnecesary and distracting verbossity when the intention is to pass the
+specialized value into another call.
+
+In conclusion, the current workarounds are too complex or verbose, especially compared
+to syntax that is consistent with the rest of the language.
 
-Undecidable Inference
-^^^^^^^^^^^^^^^^^^^^^
+Generic Specialisation
+^^^^^^^^^^^^^^^^^^^^^^
 
-There are even cases where subclass relations make type inference impossible. However,
-if you can specialise the function type checkers can infer a meaningful type.
+As in the previous example currently we can create generic aliases for different
+specialised usages:
 
 .. code-block:: python
 
-   def foo[T](x: Sequence[T] | T) -> list[T]: ...
+   NameNumberDict = dict[str, int]
+   NameNumberDict(one=1, two=2, three="3")  # Invalid: Literal["3"] is not of type int``
 
-   reveal_type(foo[bytes](b"hello"))
+This not currently possible for functions but if allowed we could easily
+specialise operations in certain sections of the codebase:
 
-Currently, type checkers do not consistently synthesise a type here.
+.. code-block:: python
+
+   def constrained_addition[T](a: T, b: T) -> T: ...
+
+   # where we work exclusively with ints
+   int_addition = constrained_addition[int]
+   int_addition(2, 4+8j)  # Invalid: complex is not of type int
+
+Unknown Types
+^^^^^^^^^^^^^
+
+Currently, it is not possible to infer the type parameters to generic functions in
+certain situations.
+
+In this example ``T`` cannot currently be meaningfully inferred, so ``x`` is
+untyped without an extra assignment:
+
+.. code-block:: python
+
+   def factory[T](func: Callable[[T], Any]) -> Foo[T]: ...
+
+   reveal_type(factory(lambda x: "Hello World" * x)) # type is Foo[Unknown]
+
+If function objects were subscriptable, however, a more specific type could be given:
+
+.. code-block:: python
+
+   reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]
 
 Unsolvable Type Parameters
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -138,7 +202,16 @@ The syntax for such a feature may look something like:
 Rationale
 ---------
 
-Function objects in this PEP is used to refer to ``FunctionType``\ , ``MethodType``\ ,
+This proposal improves the consistency of the type system, by allowing syntax that
+already looks and feels like a natural of the existing syntax for classes.
+
+If accepted, this syntax will reduce the necessity to learn about :pep:`747`\s
+``TypeForm``,  reduce verbosity and cognitive load of safely typed python.
+
+Specification
+-------------
+
+In this PEP "Function objects" is used to refer to ``FunctionType``\ , ``MethodType``\ ,
 ``BuiltinFunctionType``\ , ``BuiltinMethodType`` and ``MethodWrapperType``\ .
 
 For ``MethodType`` you should be able to write:
@@ -161,9 +234,6 @@ functions implemented in Python as possible.
 ``MethodWrapperType`` (e.g. the type of ``object().__str__``) is useful for
 generic magic methods.
 
-Specification
--------------
-
 Function objects should implement ``__getitem__`` to allow for subscription at runtime
 and return an instance of ``types.GenericAlias`` with ``__origin__`` set as the
 callable and ``__args__`` as the types passed.
@@ -201,10 +271,69 @@ The following code snippet would fail at runtime without this change as
 Interactions with ``@typing.overload``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Overloaded functions should work much the same as already, since they have no effect on
-the runtime type. The only change is that more situations will be decidable and the
-behaviour/overload can be specified by the developer rather than leaving it to ordering
-of overloads/unions.
+This PEP allows type checkers to do overloading based on type variables:
+
+.. code-block:: python
+
+   @overload
+   def serializer_for[T: str]() -> StringSerializer: ...
+   @overload
+   def serializer_for[T: list]() -> ListSerializer: ...
+
+   def serializer_for():
+       ...
+
+For overload resolution a new step will be required previous to any other, where the resolver
+will match only the overloads where the subscription may succeed.
+
+.. code-block:: python
+
+   @overload
+   def make[*Ts]() -> float: ...
+   @overload
+   def make[T]() -> int: ...
+
+   make[int]  # matches first and second overload
+   make[int, str]  # matches only first
+
+
+Functions Parameterized by ``TypeVarTuple``\ s
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Currently, type checkers disallow the use of multiple ``TypeVarTuple``\s in their
+generic parameters; however, it is currently valid to have a function as such:
+
+.. code-block:: python
+
+   def foo[*T, *U](bar: Bar[*T], baz: Baz[*U]): ...
+   def spam[*T](bar: Bar[*T]): ...
+
+This PEP does not allow functions like ``foo`` to be subscripted, for the same reason
+as defined in :pep:`PEP 646<646#multiple-type-variable-tuples-not-allowed>`, the type
+variables cannot be resolved unambiguously with the current syntax.
+
+.. code-block:: python
+
+   foo[int, str, bool, complex](Bar(), Baz())  # Invalid: cannot determine which parameters are passed to *T and *U. Explicitly parameterise the instances individually
+   spam[int, str, bool, complex](Bar())        # OK
+
+Binding Rules
+^^^^^^^^^^^^^
+Method subscription (including ``classmethods`` and ``staticmethods``), should only
+allow their function's type parameters and not the enclosing class's.
+Subscription should follow the rules specified in :pep:`PEP 696<696#binding-rules>`;
+methods should bind type parameters on attribute access.
+
+.. code-block:: python
+
+   class C[T]:
+       def method[U](self, x: T, y: U): ...
+       @classmethod
+       def cls[U](cls, x: T, y: U): ...
+
+   C[int].method[str](0, "")  # OK
+   C[int].cls[str](0, "")     # OK
+   C.cls[int, str](0, "")     # Invalid: too many type parameters
+   C.cls[str](0, "")          # OK, U will be matched to str
 
 Backwards Compatibility
 -----------------------