From e1bf79f3b0d1899c1a89673e7a78aef235104b80 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Fri, 20 Jun 2025 17:05:54 -0400 Subject: [PATCH 01/27] add draft pip --- peps/pep-9999.rst | 231 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..25e8403cfa5 --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,231 @@ +PEP: 9999 +Title: Unpacking in Comprehensions +Author: Adam Hartz , Erik Demaine +Sponsor: TBD +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 22-Jun-2025 +Python-Version: 3.15 +Post-History: `16-Oct-2021 `__ + + +Abstract +======== + +This PEP proposes extending list, set, and dictionary comprehensions, as well +as generator expressions, to allow unpacking notation (``*`` and ``**``) at the +start of the expression. For example, ``[*it for it in its]`` becomes +shorthand for ``[x for it in its for x in it]``. This notation makes it easy +to combine an arbitrary number of iterables into one list or set or generator, +or an arbitrary number of dictionaries into one dictionary. + + +Motivation +========== + +Extended unpacking notation (``*`` and ``**``) from PEP 448 makes it +easy to combine a few iterables or dictionaries:: + + [*it1, *it2, *it3] # list with the concatenation of three iterables + {*it1, *it2, *it3} # set with the union of three iterables + {**dict1, **dict2, **dict3} # dict with the combination of three dicts + +This PEP proposes extending this pattern to enable combining an arbitrary +number of iterables or dictionaries, via comprehensions:: + + [*it for it in its] # list with the concatenation of iterables in 'its' + {*it for it in its} # set with the union of iterables in 'its' + {**d for d in dicts} # dict with the combination of dicts in 'dicts' + +In addition, it defines an analogous notation for generator expressions:: + + (*it for it in its) # generator of the concatenation of iterables in 'its' + +The new notation listed above is effectively short-hand for the +following existing notation:: + + [x for it in its for x in it] + {x for it in its for x in it} + {key: value for d in dicts for key, value in d.items()} + (x for it in its for x in it) + +The new notation is more concise, avoiding the use and repetition of auxiliary +variables, and possibly more intuitive to programmers familiar with both +comprehensions and unpacking notation. The dictionary version can also be +implemented so as to be more efficient than the example above by avoiding item +tuple creation and unpacking. + +Alternatively, the notation is effectively short-hand for the following uses of +``itertools.chain``:: + + list(itertools.chain(*its)) + set(itertools.chain(*its)) + dict(itertools.chain(*(d.items() for d in dicts))) + itertools.chain(*its) + +Rationale +========= + +In the proposed notation, ``[*it for it in its]`` is analogous to +``[*its[0], *its[1], ..., *its[len(its)-1]]`` (pretending that ``its`` +is a list so supports indexing) in the same way that +``[it for it in its]`` is analogous to +``[its[0], its[1], ..., its[len(its)-1]]``. +The same analogy holds for the set, dictionary, and generator +versions. + +This proposal was motivated in part by a written exam in a Python programming +class, where several students used the notation (specifically the ``set`` +version) in their solutions, assuming that it already existed in Python. This +suggests that the notation is intuitive, even to those who are learning Python. +By contrast, the existing syntax ``[x for it in its for x in it]`` is one that +students often get wrong, with the natural impulse for many students being to +reverse the order of the ``for`` clauses. + +An example where this new notation is especially convenient (albeit suboptimal +in efficiency) is a recursive function that accumulates the set of values from +the leaves of a tree:: + + def leaf_values(node): + if node.children: + return {*leaf_values(child) for child in node.children} + else: + return {node.value} + + +Specification +============= + +In the grammar, the rules for comprehensions and generator expressions +would use ``star_named_expression`` instead of ``named_expression``:: + + listcomp[expr_ty]: + | '[' a=star_named_expression b=for_if_clauses ']' { _PyAST_ListComp(a, b, EXTRA) } + | invalid_comprehension + + setcomp[expr_ty]: + | '{' a=star_named_expression b=for_if_clauses '}' { _PyAST_SetComp(a, b, EXTRA) } + | invalid_comprehension + + genexp[expr_ty]: + | '(' a=star_named_expression b=for_if_clauses ')' { _PyAST_GeneratorExp(a, b, EXTRA) } + | invalid_comprehension + +(Small note: the current rule for ``genexp`` uses +``( assigment_expression | expression !':=')`` but this is equivalent to +``named_expression``.) + +The rule for dictionary comprehensions would be adjusted to allow a new form as well:: + + dictcomp[expr_ty]: + | '{' a=kvpair b=for_if_clauses '}' { _PyAST_DictComp(a->key, a->value, b, EXTRA) } + | '{' '**' a=expression b=for_if_clauses '}' { _PyAST_DictComp(a, NULL, b, EXTRA) } + +The grammar rule for ``invalid_comprehension`` would be changed so that using +``*`` in a comprehension no longer raises a ``SyntaxError``, and the rule for +``invalid_dict_comprehension`` (which currently only checks for ``**`` being +used in a dictionary comprehension) would be removed. + +The meaning of a starred expression in a list comprehension +``[*expr for x in it]`` is to treat each expression as an iterable, and +concatenate them, in the same way as if they were explicitly listed +via ``[*expr1, *expr2, ...]``. Similarly, ``{*expr for x in it}`` +forms a set union, as if the expressions were explicitly listed via +``{*expr1, *expr2, ...}``; and ``{**expr for x in it}`` combines +dictionaries, as if the expressions were explicitly listed via +``{**expr1, **expr2, ...}``. As usual with sets and dictionaries, +repeated elements/keys replace earlier instances. + +A generator expression ``(*expr for x in it)`` forms a generator producing +values from the concatenation of the iterables given by the expressions. +Specifically, the behavior is defined to be equivalent to the following +generator:: + + def generator(): + for x in it: + yield from expr + + +Backwards Compatibility +======================= + +In versions 3.14 and earlier, the proposed notation generates a +``SyntaxError`` (via the ``invalid_comprehension`` and +``invalid_dict_comprehension`` rules in the CPython grammar). + +The behavior of all comprehensions that are currently syntactically valid would +be unaffected by this change, so we do not anticipate much in the way of +backwards-incompatibility concerns (in principle, this change would only affect +code that relied on using unpacking operations in comprehensions raising +``SyntaxError``, which we expect to be rare). + +We do anticipate the need for additional specific error messages related to +malformed comprehensions (including, for example, using ``**`` within a list +comprehension or generator expression). For other situations, we expect the +other existent error-handling mechanisms seem to continue to work as intended. + + +How to Teach This +================= + +Previously, ``out = [...x... for x in it]`` could be thought of as +equivalent to the following code:: + + out = [] + for x in it: + out.append(...x...) + +This equivalence no longer holds when we allow ``*x`` in place of +``...x...``, because ``list.append`` accepts only a single argument. + +With the new syntax, we can instead think of +``out = [...x... for x in it]`` as equivalent to the following code [#guido]_, +regardless of whether or not ``...x...`` uses ``*``:: + + out = [] + for x in it: + out.extend([...x...]) + +Similarly, we can think of ``out = {...x... for x in it}`` as equivalent to the +following code, regardless of whether or not ``...x...`` uses ``*`` or ``**`` +or ``:``:: + + out = set() + for x in it: + out.update({...x...}) + +These examples are equivalent in the sense that the output they produce would +be the same in both the version with the comprehension and the version without +it, but note that the non-comprehension version is slightly less efficient due +to making new lists/sets/dictionaries before each ``extend`` or ``update``, which +is unnecessary in the version that uses comprehensions. + +Finally, we can think of ``out = (*...x... for x in it)`` +(specifically the version that uses a ``*``) as equivalent to the +following code:: + + def generator(): + for x in it: + yield from ...x... + out = generator() + + +Reference Implementation +======================== + +* `adqm/cpython:comprehension_unpacking `_ + + +References +========== + +.. [#guido] Message from Guido van Rossum + (https://mail.python.org/archives/list/python-ideas@python.org/message/CQPULNM6PM623PLXF5Z63BIUZGOSQEKW/) + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 3effe1fbdfd39d44049dab381f58998e0cb172f7 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 22 Jun 2025 14:11:14 -0400 Subject: [PATCH 02/27] add link to new post --- peps/pep-9999.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 25e8403cfa5..0399d5a8352 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -7,7 +7,7 @@ Type: Standards Track Content-Type: text/x-rst Created: 22-Jun-2025 Python-Version: 3.15 -Post-History: `16-Oct-2021 `__ +Post-History: `16-Oct-2021 `__, `22-Jun-2025 `__ Abstract @@ -162,8 +162,8 @@ code that relied on using unpacking operations in comprehensions raising We do anticipate the need for additional specific error messages related to malformed comprehensions (including, for example, using ``**`` within a list -comprehension or generator expression). For other situations, we expect the -other existent error-handling mechanisms seem to continue to work as intended. +comprehension or generator expression). That said, we expect a good number +of cases to be handled by the checks that are already in place. How to Teach This From f61fe2241f507005c2824e5be7d39b2e54fc7ccb Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Wed, 25 Jun 2025 11:10:48 -0400 Subject: [PATCH 03/27] reword things, reorganize, add examples --- peps/pep-9999.rst | 472 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 384 insertions(+), 88 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 0399d5a8352..4dbfb80be60 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -24,56 +24,73 @@ or an arbitrary number of dictionaries into one dictionary. Motivation ========== -Extended unpacking notation (``*`` and ``**``) from PEP 448 makes it +Extended unpacking notation (``*`` and ``**``) from :pep:`448` makes it easy to combine a few iterables or dictionaries:: [*it1, *it2, *it3] # list with the concatenation of three iterables {*it1, *it2, *it3} # set with the union of three iterables {**dict1, **dict2, **dict3} # dict with the combination of three dicts -This PEP proposes extending this pattern to enable combining an arbitrary -number of iterables or dictionaries, via comprehensions:: +But if we want to similarly combine an arbitrary number of iterables, we cannot +use unpacking in this same way. Currently, we do have a few options. We could +use explicit looping structures and built in means of combination:: - [*it for it in its] # list with the concatenation of iterables in 'its' - {*it for it in its} # set with the union of iterables in 'its' - {**d for d in dicts} # dict with the combination of dicts in 'dicts' + new_list = [] + for it in its: + new_list.extend(it) -In addition, it defines an analogous notation for generator expressions:: + new_set = set() + for it in its: + new_set.update(it) - (*it for it in its) # generator of the concatenation of iterables in 'its' + new_dict = {} + for d in dicts: + new_dict.update(d) + + def generator(): + for it in its: + yield from it -The new notation listed above is effectively short-hand for the -following existing notation:: + +Alternatively, we could be more concise by using a comprehension with two +loops:: [x for it in its for x in it] {x for it in its for x in it} {key: value for d in dicts for key, value in d.items()} (x for it in its for x in it) -The new notation is more concise, avoiding the use and repetition of auxiliary -variables, and possibly more intuitive to programmers familiar with both -comprehensions and unpacking notation. The dictionary version can also be -implemented so as to be more efficient than the example above by avoiding item -tuple creation and unpacking. - -Alternatively, the notation is effectively short-hand for the following uses of -``itertools.chain``:: +Or, we could use ``itertools.chain``:: list(itertools.chain(*its)) set(itertools.chain(*its)) dict(itertools.chain(*(d.items() for d in dicts))) itertools.chain(*its) +As an additional alternative, this PEP proposes extending the unpacking pattern +to enable the use of ``*`` and ``**`` in comprehensions and generator +expressions, for example:: + + [*it for it in its] # list with the concatenation of iterables in 'its' + {*it for it in its} # set with the union of iterables in 'its' + {**d for d in dicts} # dict with the combination of dicts in 'dicts' + (*it for it in its) # generator of the concatenation of iterables in 'its' + +This proposal also extends to asynchronous comprehensions and generator +expressions, such that, for example, ``(*x async for x in aits())`` is +equivalent to ``(x async for ait in aits() for x in ait)``. + Rationale ========= -In the proposed notation, ``[*it for it in its]`` is analogous to -``[*its[0], *its[1], ..., *its[len(its)-1]]`` (pretending that ``its`` -is a list so supports indexing) in the same way that -``[it for it in its]`` is analogous to -``[its[0], its[1], ..., its[len(its)-1]]``. -The same analogy holds for the set, dictionary, and generator -versions. +Combining iterable objects together into a single larger object is a common +task, but the options currently available for performing this operation +require levels of indirection that can make the resulting code difficult to +read and understand. + +The proposed notation is concise (avoiding the use and repetition of auxiliary +variables), and, we expect, intuitive and familiar to programmers familiar with +both comprehensions and unpacking notation. This proposal was motivated in part by a written exam in a Python programming class, where several students used the notation (specifically the ``set`` @@ -83,59 +100,79 @@ By contrast, the existing syntax ``[x for it in its for x in it]`` is one that students often get wrong, with the natural impulse for many students being to reverse the order of the ``for`` clauses. -An example where this new notation is especially convenient (albeit suboptimal -in efficiency) is a recursive function that accumulates the set of values from -the leaves of a tree:: - - def leaf_values(node): - if node.children: - return {*leaf_values(child) for child in node.children} - else: - return {node.value} +See :ref:`examples` for examples of code that could be rewritten more clearly +using the proposed syntax. Specification ============= -In the grammar, the rules for comprehensions and generator expressions -would use ``star_named_expression`` instead of ``named_expression``:: +Syntax +------ + +The necessary grammatical changes are allowing the expression in list/set +comprehensions and generator expressions to be preceded by a ``*``, and +allowing an alternative form of dictionary comprehension where the expression +is given by a single expression preceded by a ``**`` rather than a ``key: +value`` pair. + +This can be accomplished by updating the ``listcomp`` and ``setcomp`` rules to +use ``star_named_expression`` instead of ``named_expression``:: listcomp[expr_ty]: - | '[' a=star_named_expression b=for_if_clauses ']' { _PyAST_ListComp(a, b, EXTRA) } + | '[' a=star_named_expression b=for_if_clauses ']' | invalid_comprehension setcomp[expr_ty]: - | '{' a=star_named_expression b=for_if_clauses '}' { _PyAST_SetComp(a, b, EXTRA) } + | '{' a=star_named_expression b=for_if_clauses '}' | invalid_comprehension +The rule for ``genexp`` would similarly need to be modified to allow a ``starred_expression``:: + genexp[expr_ty]: - | '(' a=star_named_expression b=for_if_clauses ')' { _PyAST_GeneratorExp(a, b, EXTRA) } + | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')' | invalid_comprehension -(Small note: the current rule for ``genexp`` uses -``( assigment_expression | expression !':=')`` but this is equivalent to -``named_expression``.) - -The rule for dictionary comprehensions would be adjusted to allow a new form as well:: +The rule for dictionary comprehensions would need to be adjusted as well, to allow for this new form:: dictcomp[expr_ty]: - | '{' a=kvpair b=for_if_clauses '}' { _PyAST_DictComp(a->key, a->value, b, EXTRA) } - | '{' '**' a=expression b=for_if_clauses '}' { _PyAST_DictComp(a, NULL, b, EXTRA) } - -The grammar rule for ``invalid_comprehension`` would be changed so that using -``*`` in a comprehension no longer raises a ``SyntaxError``, and the rule for -``invalid_dict_comprehension`` (which currently only checks for ``**`` being -used in a dictionary comprehension) would be removed. - -The meaning of a starred expression in a list comprehension -``[*expr for x in it]`` is to treat each expression as an iterable, and -concatenate them, in the same way as if they were explicitly listed -via ``[*expr1, *expr2, ...]``. Similarly, ``{*expr for x in it}`` -forms a set union, as if the expressions were explicitly listed via -``{*expr1, *expr2, ...}``; and ``{**expr for x in it}`` combines -dictionaries, as if the expressions were explicitly listed via -``{**expr1, **expr2, ...}``. As usual with sets and dictionaries, -repeated elements/keys replace earlier instances. + | '{' a=kvpair b=for_if_clauses '}' + | '{' '**' a=bitwise_or b=for_if_clauses '}' + +We propose no additional changes to the way that argument unpacking for +function calls is handled, i.e., we propose retaining the rule that generator +expressions provided as the sole argument to functions do not require +additional redundant parentheses, i.e., that ``f(*x for x in it)`` should be +equivalent to ``f((*x for x in it))`` (see :ref:`functionargs` for more discussion). + + +Semantics: List/Set/Dict Comprehensions +--------------------------------------- + +The meaning of a starred expression in a list comprehension ``[*expr for x in +it]`` is to treat each expression as an iterable, and concatenate them, in the +same way as if they were explicitly listed via ``[*expr1, *expr2, ...]``. +Similarly, ``{*expr for x in it}`` forms a set union, as if the expressions +were explicitly listed via ``{*expr1, *expr2, ...}``; and ``{**expr for x in +it}`` combines dictionaries, as if the expressions were explicitly listed via +``{**expr1, **expr2, ...}``, retaining all of the equivalent semantics for +combining collections in this way (e.g., later values replacing earlier values +associated with the same key when combining dictionaries). + +For list and set comprehensions, the generated bytecode between the starred and +unstarred version of the same comprehension should be identical, except for +replacing the opcode for adding a single element to the collection being built +up (``LIST_APPEND`` and ``SET_ADD``, respectively) with the opcode for +combining collections of that type (``LIST_EXTEND`` and ``SET_UPDATE``, +respectively). + +Dictionary comprehensions should follow a similar pattern. The resulting +bytecode will necessarily be somewhat different, but the key difference will be +the use of ``DICT_UPDATE`` instead of ``MAP_ADD`` as the way to add elements to +the new dictionary. + +Semantics: Generator Expressions +-------------------------------- A generator expression ``(*expr for x in it)`` forms a generator producing values from the concatenation of the iterables given by the expressions. @@ -146,42 +183,219 @@ generator:: for x in it: yield from expr +For synchronous generator expressions, the generated bytecode for the starred +and unstarred version of the same generator expression should be similar, but +the starred expression should using ``YIELD_FROM`` instead of ``YIELD_VALUE`` +inside the loop. + +For async generator expressions, ``(*expr async for x in ait())``, the equivalence +is instead:: + + async def generator(): + async for x in ait(): + for i in expr: + yield i + +For async generator expressions, which don't allow the internal use of +``YIELD_FROM``, we instead propose mimicking the functionality of the existing +``(z async for x in y for z in x)`` syntax more directly. The resulting +bytecode for, for example, ``(*x for x in y)`` should be the same as the +bytecode for ``(z async for x in y for z in x)``, with the natural exception of +the ``STORE_FAST_LOAD_FAST`` used to bind the variable ``z``. + +For generator expressions that make use of the walrus operator ``:=`` from +:pep:`572`, note that we are not proposing changing the order of evaluation of +the various pieces of the comprehension, nor the rules around scoping. So, for +example, in the expression ``(*(y := [i, i+1]) for i in (0, 2, 4))``, ``y`` +will be defined (in the containing scope) as ``[0, 1]`` until just before the +resulting generator produces its third value, at which point the expression is +evaluated for its second time. + +Error Reporting +--------------- + +Currently, the proposed syntax generates a ``SyntaxError`` (via the +``invalid_comprehension`` and ``invalid_dict_comprehension`` rules). + +Allowing these forms to be recognized as syntactically valid requires changing +the grammar rule for ``invalid_comprehension`` so that using ``*`` in a +comprehension no longer raises a ``SyntaxError``, as well as removing the rule +for ``invalid_dict_comprehension`` (which currently only checks for ``**`` +being used in a dictionary comprehension). + +We also propose additional specific error messages in the following cases: + +* Attempting to use ``**`` in a list comprehension or generator expression + should report that dictionary unpacking cannot be used in those structures:: + + invalid_comprehension: + | '[' a='**' b=expression for_if_clauses { + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "dict unpacking cannot be used in list comprehension") } + | '(' a='**' b=expression for_if_clauses { + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "dict unpacking cannot be used in generator expression") } + ... + + +* The existing error error message for attempting to use ``*`` in a dictionary + value should be retained, but we also propose reporting similar messages + when attempting to use ``*`` or ``**`` unpacking on a dictionary key or value:: + + invalid_double_starred_kvpairs: + ... + | a='*' b=bitwise_or ':' expression { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use a starred expression in a dictionary key") } + | a='**' b=bitwise_or ':' expression { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use dict unpacking in a dictionary key") } + | expression ':' a='*' b=bitwise_or { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use a starred expression in a dictionary value") } + | expression ':' a='**' b=bitwise_or { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use dict unpacking in a dictionary value") } + ... + +.. _reference: + +Reference Implementation +======================== + +A reference implementation is available at +`adqm/cpython:comprehension_unpacking +`_. Backwards Compatibility ======================= -In versions 3.14 and earlier, the proposed notation generates a -``SyntaxError`` (via the ``invalid_comprehension`` and -``invalid_dict_comprehension`` rules in the CPython grammar). - The behavior of all comprehensions that are currently syntactically valid would be unaffected by this change, so we do not anticipate much in the way of backwards-incompatibility concerns (in principle, this change would only affect code that relied on using unpacking operations in comprehensions raising ``SyntaxError``, which we expect to be rare). -We do anticipate the need for additional specific error messages related to -malformed comprehensions (including, for example, using ``**`` within a list -comprehension or generator expression). That said, we expect a good number -of cases to be handled by the checks that are already in place. +Rejected Alternative Proposals +============================== + +The primary goal when thinking through the specification above was consistency +with existing norms around unpacking and comprehensions / generator +expressions. One way to interpret this is that the goal was to write the +specification so as to require the smallest possible change(s) to the existing +grammar and code generation and letting the existing code inform the surrounding +semantics. + +Below we discuss some of the common concerns/alternative proposals that have +come up in dicussions in the past but that are not included in this proposal. + +.. _functionargs: + +Starred Generators as Function Arguments +---------------------------------------- + +One common concern that has arisen multiple times (not only in our discussions +but also in previous discussions around this same idea) is a possible +syntactical ambiguity when passing a starred generator as the sole argument to +``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as +a reason for not including a similar generalization as part of the proposal. + +Our proposal is that ``f(*x for x in y)`` should be interpreted as ``f((*x for +x in y))`` and not attempt unpacking, but several alternatives were suggested +(or have been suggested) in the past, including: + +* interpreting ``f(*x for x in y)`` as ``f(*(x for x in y)``, +* interpreting ``f(*x for x in y)`` as ``f(*(*x for x in y))``, or +* continuing to raise a ``SyntaxError`` for ``f(*x for x in y)`` even if the + other aspects of this proposal are accepted. + +The reason to prefer this proposal over these alternatives is the preservation +of existent conventions for punctuation around generator expressions. +Currently, the general rule is that generator expressions must be wrapped in +parentheses except when provided as the sole argument to a function, and we opt +for maintaining that rule even as we allow more kinds of generator expressions. +This option maintains a full symmetry between comprehensions and generator +expressions that use unpacking and those that don't. + +Currently, we have the following conventions:: + + f([x for x in y]) # pass in a single list + f({x for x in y}) # pass in a single set + f(x for x in y) # pass in a single generator (no additional parentheses required around genexp) + + f(*[x for x in y]) # pass in elements from the list separately + f(*{x for x in y}) # pass in elements from the set separately + f(*(x for x in y)) # pass in elements from the generator separately (parentheses required) + +This proposal opts to maintains those conventions even when the comprehensions +make use of unpacking:: + + f([*x for x in y]) # pass in a single list + f({*x for x in y}) # pass in a single set + f(*x for x in y) # pass in a single generator (no additional parentheses required around genexp) + + f(*[*x for x in y]) # pass in elements from the list separately + f(*{*x for x in y}) # pass in elements from the set separately + f(*(*x for x in y)) # pass in elements from the generator separately (parentheses required) How to Teach This ================= -Previously, ``out = [...x... for x in it]`` could be thought of as -equivalent to the following code:: +Currently, a common way to introduce the notion of comprehensions (which is +employed by the Python Tutorial) is to demonstrate equivalent code. For +example, this method would say that, for example, ``out = [expr for x in it]`` +is equivalent to the following code:: + + out = [] + for x in it: + out.append(expr) + +Taking this approach, we can introduce ``out = [*expr for x in it]`` as instead +being equivalent to the following (which uses ``extend`` instead of +``append``:: out = [] for x in it: - out.append(...x...) + out.extend(expr) + +Set and dict comprehensions that make use of unpacking can also be introduced +by a similar analogy:: + + # equivalent to out = {expr for x in it} + out = set() + for x in it: + out.add(expr) -This equivalence no longer holds when we allow ``*x`` in place of -``...x...``, because ``list.append`` accepts only a single argument. + # equivalent to out = {*expr for x in it} + out = set() + for x in it: + out.update(expr) + + # equivalent to out = {k_expr: v_expr for x in it} + out = {} + for x in it: + out[k_expr] = v_expr + + # equivalent to out = {**expr for x in it} + out = {} + for x in it: + out.update(expr) + +And we can take a similar approach to illustrate the behavior of generator +expressions that involve unpacking:: + + # equivalent to g = (expr for x in it) + def generator(): + for x in it: + yield expr + g = generator() + + # equivalent to g = (*expr for x in it) + def generator(): + for x in it: + yield from expr + +We can then generalize from these specific examples to the idea that, +wherever a non-starred comprehension/genexp would use an operator that +adds a single element to a collection, the starred would instead use +an operator that adds multiple elements to that collection. -With the new syntax, we can instead think of -``out = [...x... for x in it]`` as equivalent to the following code [#guido]_, -regardless of whether or not ``...x...`` uses ``*``:: + +Alternatively, we don't need to think of the two ideas as separate; instead, +with the new syntax, we can instead think of ``out = [...x... for x in it]`` as +equivalent to the following code [#guido]_, regardless of whether or not +``...x...`` uses ``*``:: out = [] for x in it: @@ -201,20 +415,103 @@ it, but note that the non-comprehension version is slightly less efficient due to making new lists/sets/dictionaries before each ``extend`` or ``update``, which is unnecessary in the version that uses comprehensions. -Finally, we can think of ``out = (*...x... for x in it)`` -(specifically the version that uses a ``*``) as equivalent to the -following code:: +.. _examples: - def generator(): - for x in it: - yield from ...x... - out = generator() +Code Examples +============= +This section shows some illustrative examples of how small pieces of code from +the standard library could be rewritten to make use of this new syntax to +improve consision and readability. The :ref:`reference` continues to pass all +tests with these replacements made. -Reference Implementation -======================== +Replacing from_iterable and Friends +----------------------------------- + +While not always the right choice, replacing ``itertools.chain.from_iterable`` +and ``map`` can avoid an extra level of redirection, resulting in code that +follows conventional wisdom that comprehensions are more readable than +map/filter. + +* From ``dataclasses.py``:: + + # current: + inherited_slots = set( + itertools.chain.from_iterable(map(_get_slots, cls.__mro__[1:-1])) + ) -* `adqm/cpython:comprehension_unpacking `_ + # improved: + inherited_slots = {*_get_slots(c) for c in cls.__mro__[1:-1]} + +* From ``importlib/metadata/__init__.py``:: + + # current: + return itertools.chain.from_iterable( + path.search(prepared) for path in map(FastPath, paths) + ) + + # improved: + return (*FastPath(path).search(prepared) for path in paths) + +* From ``collections/__init__.py`` (``Counter`` class):: + + # current: + return _chain.from_iterable(_starmap(_repeat, self.items())) + + # improved: + return (*_repeat(elt, num) for elt, num in self.items()) + +* From ``zipfile/_path/__init__.py``:: + + # current: + parents = itertools.chain.from_iterable(map(_parents, names)) + + # improved: + parents = (*_parents(name) for name in names) + +* From ``_pyrepl/_module_completer.py``:: + + # current: + search_locations = set(chain.from_iterable( + getattr(spec, 'submodule_search_locations', []) + for spec in specs if spec + )) + + # improved: + search_locations = { + *getattr(spec, 'submodule_search_locations', []) + for spec in specs if spec + } + +Replacing Double Loops in Comprehensions +---------------------------------------- + +Replacing double loops in comprehensions avoids the need for defining and +referencing an auxiliary variable, reducing clutter. + +* From ``multiprocessing.py``:: + + # current: + children = (child for path in self._paths for child in path.iterdir()) + + # improved: + children = (*path.iterdir() for path in self._paths) + +* From ``Lib/asyncio/base_events.py``:: + + # current: + exceptions = [exc for sub in exceptions for exc in sub] + + # improved: + exceptions = [*sub for sub in exceptions] + +* From ``_weakrefset.py``:: + + # current: + return self.__class__(e for s in (self, other) for e in s) + + # improved: + return self.__class__(*s for s in (self, other)) References @@ -223,7 +520,6 @@ References .. [#guido] Message from Guido van Rossum (https://mail.python.org/archives/list/python-ideas@python.org/message/CQPULNM6PM623PLXF5Z63BIUZGOSQEKW/) - Copyright ========= From 8abeb2123c336fb2196b0e58b08a2bdd70c694b3 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Wed, 25 Jun 2025 14:12:54 -0400 Subject: [PATCH 04/27] phrasing of section on async generators --- peps/pep-9999.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 4dbfb80be60..54c021fd4bf 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -189,19 +189,19 @@ the starred expression should using ``YIELD_FROM`` instead of ``YIELD_VALUE`` inside the loop. For async generator expressions, ``(*expr async for x in ait())``, the equivalence -is instead:: +is more like the following:: async def generator(): async for x in ait(): for i in expr: yield i -For async generator expressions, which don't allow the internal use of -``YIELD_FROM``, we instead propose mimicking the functionality of the existing -``(z async for x in y for z in x)`` syntax more directly. The resulting -bytecode for, for example, ``(*x for x in y)`` should be the same as the -bytecode for ``(z async for x in y for z in x)``, with the natural exception of -the ``STORE_FAST_LOAD_FAST`` used to bind the variable ``z``. +Since ``YIELD_FROM`` is not allowed inside of async generators, we instead +propose mimicking the functionality of the existing ``(z async for x in y for z +in x)`` syntax more directly. The resulting bytecode for, for example, ``(*x +for x in y)`` should be the same as the bytecode for ``(z async for x in y for +z in x)``, with the natural exception of the ``STORE_FAST_LOAD_FAST`` used to +bind the variable ``z``. For generator expressions that make use of the walrus operator ``:=`` from :pep:`572`, note that we are not proposing changing the order of evaluation of From be81afefee924b413cb956f4ce44ab2da3ae9df6 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Wed, 25 Jun 2025 15:54:43 -0400 Subject: [PATCH 05/27] add missing line from example --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 54c021fd4bf..007fb054f6c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -385,13 +385,13 @@ expressions that involve unpacking:: def generator(): for x in it: yield from expr + g = generator() We can then generalize from these specific examples to the idea that, wherever a non-starred comprehension/genexp would use an operator that adds a single element to a collection, the starred would instead use an operator that adds multiple elements to that collection. - Alternatively, we don't need to think of the two ideas as separate; instead, with the new syntax, we can instead think of ``out = [...x... for x in it]`` as equivalent to the following code [#guido]_, regardless of whether or not From 1b74bfa60ac56c6614e42a321a614e8b9fcc904a Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 28 Jun 2025 02:26:22 -0400 Subject: [PATCH 06/27] reorder things --- peps/pep-9999.rst | 123 +++++++++++++++++++++++----------------------- 1 file changed, 61 insertions(+), 62 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 007fb054f6c..ae0a5ca2e3c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -266,68 +266,6 @@ backwards-incompatibility concerns (in principle, this change would only affect code that relied on using unpacking operations in comprehensions raising ``SyntaxError``, which we expect to be rare). -Rejected Alternative Proposals -============================== - -The primary goal when thinking through the specification above was consistency -with existing norms around unpacking and comprehensions / generator -expressions. One way to interpret this is that the goal was to write the -specification so as to require the smallest possible change(s) to the existing -grammar and code generation and letting the existing code inform the surrounding -semantics. - -Below we discuss some of the common concerns/alternative proposals that have -come up in dicussions in the past but that are not included in this proposal. - -.. _functionargs: - -Starred Generators as Function Arguments ----------------------------------------- - -One common concern that has arisen multiple times (not only in our discussions -but also in previous discussions around this same idea) is a possible -syntactical ambiguity when passing a starred generator as the sole argument to -``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as -a reason for not including a similar generalization as part of the proposal. - -Our proposal is that ``f(*x for x in y)`` should be interpreted as ``f((*x for -x in y))`` and not attempt unpacking, but several alternatives were suggested -(or have been suggested) in the past, including: - -* interpreting ``f(*x for x in y)`` as ``f(*(x for x in y)``, -* interpreting ``f(*x for x in y)`` as ``f(*(*x for x in y))``, or -* continuing to raise a ``SyntaxError`` for ``f(*x for x in y)`` even if the - other aspects of this proposal are accepted. - -The reason to prefer this proposal over these alternatives is the preservation -of existent conventions for punctuation around generator expressions. -Currently, the general rule is that generator expressions must be wrapped in -parentheses except when provided as the sole argument to a function, and we opt -for maintaining that rule even as we allow more kinds of generator expressions. -This option maintains a full symmetry between comprehensions and generator -expressions that use unpacking and those that don't. - -Currently, we have the following conventions:: - - f([x for x in y]) # pass in a single list - f({x for x in y}) # pass in a single set - f(x for x in y) # pass in a single generator (no additional parentheses required around genexp) - - f(*[x for x in y]) # pass in elements from the list separately - f(*{x for x in y}) # pass in elements from the set separately - f(*(x for x in y)) # pass in elements from the generator separately (parentheses required) - -This proposal opts to maintains those conventions even when the comprehensions -make use of unpacking:: - - f([*x for x in y]) # pass in a single list - f({*x for x in y}) # pass in a single set - f(*x for x in y) # pass in a single generator (no additional parentheses required around genexp) - - f(*[*x for x in y]) # pass in elements from the list separately - f(*{*x for x in y}) # pass in elements from the set separately - f(*(*x for x in y)) # pass in elements from the generator separately (parentheses required) - How to Teach This ================= @@ -513,6 +451,67 @@ referencing an auxiliary variable, reducing clutter. # improved: return self.__class__(*s for s in (self, other)) +Rejected Alternative Proposals +============================== + +The primary goal when thinking through the specification above was consistency +with existing norms around unpacking and comprehensions / generator +expressions. One way to interpret this is that the goal was to write the +specification so as to require the smallest possible change(s) to the existing +grammar and code generation and letting the existing code inform the surrounding +semantics. + +Below we discuss some of the common concerns/alternative proposals that have +come up in dicussions in the past but that are not included in this proposal. + +.. _functionargs: + +Starred Generators as Function Arguments +---------------------------------------- + +One common concern that has arisen multiple times (not only in our discussions +but also in previous discussions around this same idea) is a possible +syntactical ambiguity when passing a starred generator as the sole argument to +``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as +a reason for not including a similar generalization as part of the proposal. + +Our proposal is that ``f(*x for x in y)`` should be interpreted as ``f((*x for +x in y))`` and not attempt unpacking, but several alternatives were suggested +(or have been suggested) in the past, including: + +* interpreting ``f(*x for x in y)`` as ``f(*(x for x in y)``, +* interpreting ``f(*x for x in y)`` as ``f(*(*x for x in y))``, or +* continuing to raise a ``SyntaxError`` for ``f(*x for x in y)`` even if the + other aspects of this proposal are accepted. + +The reason to prefer this proposal over these alternatives is the preservation +of existent conventions for punctuation around generator expressions. +Currently, the general rule is that generator expressions must be wrapped in +parentheses except when provided as the sole argument to a function, and we opt +for maintaining that rule even as we allow more kinds of generator expressions. +This option maintains a full symmetry between comprehensions and generator +expressions that use unpacking and those that don't. + +Currently, we have the following conventions:: + + f([x for x in y]) # pass in a single list + f({x for x in y}) # pass in a single set + f(x for x in y) # pass in a single generator (no additional parentheses required around genexp) + + f(*[x for x in y]) # pass in elements from the list separately + f(*{x for x in y}) # pass in elements from the set separately + f(*(x for x in y)) # pass in elements from the generator separately (parentheses required) + +This proposal opts to maintains those conventions even when the comprehensions +make use of unpacking:: + + f([*x for x in y]) # pass in a single list + f({*x for x in y}) # pass in a single set + f(*x for x in y) # pass in a single generator (no additional parentheses required around genexp) + + f(*[*x for x in y]) # pass in elements from the list separately + f(*{*x for x in y}) # pass in elements from the set separately + f(*(*x for x in y)) # pass in elements from the generator separately (parentheses required) References ========== From e776a63d00e942b54fa03b223f15c9a358abf152 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 29 Jun 2025 02:27:58 -0400 Subject: [PATCH 07/27] rephrasing, adding more examples and discussion --- peps/pep-9999.rst | 273 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 208 insertions(+), 65 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index ae0a5ca2e3c..06f54532bcf 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -15,11 +15,14 @@ Abstract This PEP proposes extending list, set, and dictionary comprehensions, as well as generator expressions, to allow unpacking notation (``*`` and ``**``) at the -start of the expression. For example, ``[*it for it in its]`` becomes -shorthand for ``[x for it in its for x in it]``. This notation makes it easy -to combine an arbitrary number of iterables into one list or set or generator, -or an arbitrary number of dictionaries into one dictionary. +start of the expression, providing a concise way of combining an arbitrary +number of iterables into one list or set or generator, or an arbitrary number +of dictionaries into one dictionary, for example:: + [*it for it in its] # list with the concatenation of iterables in 'its' + {*it for it in its} # set with the union of iterables in 'its' + {**d for d in dicts} # dict with the combination of dicts in 'dicts' + (*it for it in its) # generator of the concatenation of iterables in 'its' Motivation ========== @@ -67,6 +70,12 @@ Or, we could use ``itertools.chain``:: dict(itertools.chain(*(d.items() for d in dicts))) itertools.chain(*its) +Or, for all but the generator, we could use ``functools.reduce``:: + + functools.reduce(operator.iconcat, its, (new_list := [])) + functools.reduce(operator.ior, its, (new_set := set())) + functools.reduce(operator.ior, its, (new_dict := {})) + As an additional alternative, this PEP proposes extending the unpacking pattern to enable the use of ``*`` and ``**`` in comprehensions and generator expressions, for example:: @@ -84,12 +93,15 @@ Rationale ========= Combining iterable objects together into a single larger object is a common -task, but the options currently available for performing this operation -require levels of indirection that can make the resulting code difficult to -read and understand. +task, as evidenced by, for example, `this StackOverflow post +`_ +asking about flattening a list of lists, which has been viewed 4.6 million +times. Despite this being a common task, the options currently available for +performing this operation concisely require levels of indirection that can make +the resulting code difficult to read and understand. The proposed notation is concise (avoiding the use and repetition of auxiliary -variables), and, we expect, intuitive and familiar to programmers familiar with +variables) and, we expect, intuitive and familiar to programmers familiar with both comprehensions and unpacking notation. This proposal was motivated in part by a written exam in a Python programming @@ -101,7 +113,7 @@ students often get wrong, with the natural impulse for many students being to reverse the order of the ``for`` clauses. See :ref:`examples` for examples of code that could be rewritten more clearly -using the proposed syntax. +and concisely using the proposed syntax. Specification @@ -133,17 +145,21 @@ The rule for ``genexp`` would similarly need to be modified to allow a ``starred | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')' | invalid_comprehension -The rule for dictionary comprehensions would need to be adjusted as well, to allow for this new form:: +The rule for dictionary comprehensions would need to be adjusted as well, to +allow for this new form:: dictcomp[expr_ty]: - | '{' a=kvpair b=for_if_clauses '}' - | '{' '**' a=bitwise_or b=for_if_clauses '}' + | '{' a=double_starred_kvpair b=for_if_clauses '}' + +No change should be made to the way that argument unpacking is handled in +function calls, i.e., the general rule that generator expressions provided as +the sole argument to functions do not require additional redundant parentheses +should be retained. Note that this implies that, for example, ``f(*x for x in +it)`` is equivalent to ``f((*x for x in it))`` (see :ref:`functionargs` for +more discussion). -We propose no additional changes to the way that argument unpacking for -function calls is handled, i.e., we propose retaining the rule that generator -expressions provided as the sole argument to functions do not require -additional redundant parentheses, i.e., that ``f(*x for x in it)`` should be -equivalent to ``f((*x for x in it))`` (see :ref:`functionargs` for more discussion). +``*`` and ``**`` should only be allowed at the top-most level of the expression +in the comprehension (see :ref:`moregeneral` for more discussion). Semantics: List/Set/Dict Comprehensions @@ -188,28 +204,63 @@ and unstarred version of the same generator expression should be similar, but the starred expression should using ``YIELD_FROM`` instead of ``YIELD_VALUE`` inside the loop. -For async generator expressions, ``(*expr async for x in ait())``, the equivalence -is more like the following:: +For async generator expressions, ``(*expr async for x in ait())``, the +equivalence is more like the following:: async def generator(): async for x in ait(): for i in expr: yield i -Since ``YIELD_FROM`` is not allowed inside of async generators, we instead -propose mimicking the functionality of the existing ``(z async for x in y for z -in x)`` syntax more directly. The resulting bytecode for, for example, ``(*x -for x in y)`` should be the same as the bytecode for ``(z async for x in y for -z in x)``, with the natural exception of the ``STORE_FAST_LOAD_FAST`` used to -bind the variable ``z``. - -For generator expressions that make use of the walrus operator ``:=`` from -:pep:`572`, note that we are not proposing changing the order of evaluation of -the various pieces of the comprehension, nor the rules around scoping. So, for -example, in the expression ``(*(y := [i, i+1]) for i in (0, 2, 4))``, ``y`` -will be defined (in the containing scope) as ``[0, 1]`` until just before the -resulting generator produces its third value, at which point the expression is -evaluated for its second time. +Since ``YIELD_FROM`` is not allowed inside of async generators, this form +should instead mimic the functionality of the existing ``(z async for x in y +for z in x)`` syntax more directly. The resulting bytecode for, for example, +``(*x for x in y)`` should be the same as the bytecode for ``(z async for x in +y for z in x)``, with the natural exception of the ``STORE_FAST_LOAD_FAST`` +used to bind the variable ``z``. + +Interaction with Assignment Expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Note that this proposal does not suggest changing the order of evaluation of +the various pieces of the comprehension, nor any rules about scoping. This is +particularly relevant for generator expressions that make use of the "walrus +operator" ``:=`` from :pep:`572`, which, when used in a comprehension or a +generator expression, performs its variable binding in the containing scope +rather than locally to the comprehension. + +As an example, consider the generator that results from evaluating the +expression ``(*(y := [i, i+1]) for i in (0, 2, 4))``. This is approximately +equivalent to the following generator, except that in its generator expression +form, ``y`` will be bound in the containing scope instead of locally:: + + def generator(): + for i in (0, 2, 4): + yield from (y := [i, i+1]) + +In this example, the subexpression ``(y := [i, i+1])`` is evaluated exactly +three times before the generator is exhausted: just after assigning ``i`` in +the comprehension to ``0``, ``2``, and ``4``, respectively. Thus, ``y`` (in +the containing scope) will be modified at those points in time:: + + >>> g = (*(y := [i, i+1]) for i in (0, 2, 4)) + >>> y + Traceback (most recent call last): + File "", line 1, in + y + NameError: name 'y' is not defined + >>> next(g) + 0 + >>> y + [0, 1] + >>> next(g) + 1 + >>> y + [0, 1] + >>> next(g) + 2 + >>> y + [2, 3] Error Reporting --------------- @@ -223,48 +274,104 @@ comprehension no longer raises a ``SyntaxError``, as well as removing the rule for ``invalid_dict_comprehension`` (which currently only checks for ``**`` being used in a dictionary comprehension). -We also propose additional specific error messages in the following cases: +Additional specific error messages should be provided in at least the following +cases: * Attempting to use ``**`` in a list comprehension or generator expression - should report that dictionary unpacking cannot be used in those structures:: - - invalid_comprehension: - | '[' a='**' b=expression for_if_clauses { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "dict unpacking cannot be used in list comprehension") } - | '(' a='**' b=expression for_if_clauses { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "dict unpacking cannot be used in generator expression") } - ... - + should report that dictionary unpacking cannot be used in those structures, + for example:: + + >>> [**x for x in y] + File "", line 1 + [**x for x in y] + ^^^ + SyntaxError: cannot use dict unpacking in list comprehension + + >>> (**x for x in y) + File "", line 1 + (**x for x in y) + ^^^ + SyntaxError: cannot use dict unpacking in generator expression + + +* The existing error message for attempting to use ``*`` in a dictionary + key/value should be retained, but similar messages should be reported + when attempting to use ``**`` unpacking on a dictionary key or value, for + example:: + + >>> {*k: v for k,v in items} + File "", line 1 + {*k: v for k,v in items} + ^^ + SyntaxError: cannot use a starred expression in a dictionary key + + >>> {k: *v for k,v in items} + File "", line 1 + {k: *v for k,v in items} + ^^ + SyntaxError: cannot use a starred expression in a dictionary value + + >>> {**k: v for k,v in items} + File "", line 1 + {**k: v for k,v in items} + ^^^ + SyntaxError: cannot use dict unpacking in a dictionary key + + >>> {k: **v for k,v in items} + File "", line 1 + {k: **v for k,v in items} + ^^^ + SyntaxError: cannot use dict unpacking in a dictionary value + +* The phrasing of some other existing error messages should similarly be + adjusted to account for the presence of the new syntax, and/or to clarify + ambiguous or confusing cases relating to unpacking more generally + (particularly those mentioned in :ref:`moregeneral`), for example:: + + >>> [*x if x else y] + File "", line 1 + [*x if x else y] + ^^^^^^^^^^^^^^ + SyntaxError: invalid starred expression. did you forget to wrap the conditional expression in parentheses? + + >>> {**x if x else y} + File "", line 1 + {**x if x else y} + ^^^^^^^^^^^^^^^ + SyntaxError: invalid double starred expression. did you forget to wrap the conditional expression in parentheses? + + >>> [x if x else *y] + File "", line 1 + [x if x else *y] + ^ + SyntaxError: cannot unpack only part of a conditional expression + + >>> {x if x else **y} + File "", line 1 + {x if x else **y} + ^^ + SyntaxError: cannot use dict unpacking on only part of a conditional expression -* The existing error error message for attempting to use ``*`` in a dictionary - value should be retained, but we also propose reporting similar messages - when attempting to use ``*`` or ``**`` unpacking on a dictionary key or value:: - - invalid_double_starred_kvpairs: - ... - | a='*' b=bitwise_or ':' expression { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use a starred expression in a dictionary key") } - | a='**' b=bitwise_or ':' expression { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use dict unpacking in a dictionary key") } - | expression ':' a='*' b=bitwise_or { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use a starred expression in a dictionary value") } - | expression ':' a='**' b=bitwise_or { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot use dict unpacking in a dictionary value") } - ... .. _reference: Reference Implementation ======================== -A reference implementation is available at -`adqm/cpython:comprehension_unpacking -`_. +A `reference implementation `_ +is available, which implements this functionality, including draft documentation and +additional test cases. Backwards Compatibility ======================= The behavior of all comprehensions that are currently syntactically valid would be unaffected by this change, so we do not anticipate much in the way of -backwards-incompatibility concerns (in principle, this change would only affect -code that relied on using unpacking operations in comprehensions raising -``SyntaxError``, which we expect to be rare). +backwards-incompatibility concerns. In principle, this change would only +affect code that relied on the fact that attempting to use unpacking operations +in comprehensions would raise a ``SyntaxError``, or who relied on the +particular phrasing of any of the old error messages being replaced, which we +expect to be rare. How to Teach This @@ -331,7 +438,7 @@ adds a single element to a collection, the starred would instead use an operator that adds multiple elements to that collection. Alternatively, we don't need to think of the two ideas as separate; instead, -with the new syntax, we can instead think of ``out = [...x... for x in it]`` as +with the new syntax, we can think of ``out = [...x... for x in it]`` as equivalent to the following code [#guido]_, regardless of whether or not ``...x...`` uses ``*``:: @@ -350,8 +457,8 @@ or ``:``:: These examples are equivalent in the sense that the output they produce would be the same in both the version with the comprehension and the version without it, but note that the non-comprehension version is slightly less efficient due -to making new lists/sets/dictionaries before each ``extend`` or ``update``, which -is unnecessary in the version that uses comprehensions. +to making new lists/sets/dictionaries before each ``extend`` or ``update``, +which is unnecessary in the version that uses comprehensions. .. _examples: @@ -451,6 +558,14 @@ referencing an auxiliary variable, reducing clutter. # improved: return self.__class__(*s for s in (self, other)) + +Disadvantages +============= + +.. note:: + TODO: add a summary of negative feedback and confusing points/examples from + the discourse thread. + Rejected Alternative Proposals ============================== @@ -513,6 +628,34 @@ make use of unpacking:: f(*{*x for x in y}) # pass in elements from the set separately f(*(*x for x in y)) # pass in elements from the generator separately (parentheses required) +.. _moregeneral: + +Further Generalizing Unpacking Operators +---------------------------------------- + +Another suggestion that came out of our discussion involved further +generalizing the ``*`` beyond simply allowing it to be used to unpack the +expression in a comprehension. Two main flavors of this extension were +considered: + +* making ``*`` and ``**`` true unary operators that create a new kind of + ``Unpackable`` object (or similar), which comprehensions could treat by + unpacking it but which could also be used in other contexts; or + +* continuing to allow ``*`` and ``**`` only in the places they are allowed + elsewhere in this proposal (expression lists, comprehensions, generator + expressions, and argument lists), but also allow them to be used in + subexpressions within a comprehension, allowing, for example, the following + as a way to flatten a list that contains some iterables but some non-iterable + objects:: + + [*x if isinstance(x, Iterable) else x for x in [[1,2,3], 4]] + +Because these variants were deemed to be substantially more complex (both to +understand and to implement) and of only marginal utility, neither is included +in this PEP. As such, these forms should continue to raise a ``SyntaxError``, +but with a new error message as described above. + References ========== @@ -522,5 +665,5 @@ References Copyright ========= -This document is placed in the public domain or under the -CC0-1.0-Universal license, whichever is more permissive. +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. From 17b57891419c148a3c75b9d895b490a8d6d03d10 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Mon, 30 Jun 2025 12:49:53 -0400 Subject: [PATCH 08/27] add disadvantages section, remove 'we' --- peps/pep-9999.rst | 78 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 24 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 06f54532bcf..e6eb1be5aff 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -5,7 +5,7 @@ Sponsor: TBD Status: Draft Type: Standards Track Content-Type: text/x-rst -Created: 22-Jun-2025 +Created: 30-Jun-2025 Python-Version: 3.15 Post-History: `16-Oct-2021 `__, `22-Jun-2025 `__ @@ -93,9 +93,9 @@ Rationale ========= Combining iterable objects together into a single larger object is a common -task, as evidenced by, for example, `this StackOverflow post +task. One `StackOverflow post `_ -asking about flattening a list of lists, which has been viewed 4.6 million +asking about flattening a list of lists, for example, has been viewed 4.6 million times. Despite this being a common task, the options currently available for performing this operation concisely require levels of indirection that can make the resulting code difficult to read and understand. @@ -558,14 +558,6 @@ referencing an auxiliary variable, reducing clutter. # improved: return self.__class__(*s for s in (self, other)) - -Disadvantages -============= - -.. note:: - TODO: add a summary of negative feedback and confusing points/examples from - the discourse thread. - Rejected Alternative Proposals ============================== @@ -584,15 +576,17 @@ come up in dicussions in the past but that are not included in this proposal. Starred Generators as Function Arguments ---------------------------------------- -One common concern that has arisen multiple times (not only in our discussions -but also in previous discussions around this same idea) is a possible -syntactical ambiguity when passing a starred generator as the sole argument to -``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as -a reason for not including a similar generalization as part of the proposal. +One common concern that has arisen multiple times (not only in the discussion +threads linked above but also in previous discussions around this same idea) is +a possible syntactical ambiguity when passing a starred generator as the sole +argument to ``f(*x for x in y)``. In the original :pep:`448`, this ambiguity +was cited as a reason for not including a similar generalization as part of the +proposal. -Our proposal is that ``f(*x for x in y)`` should be interpreted as ``f((*x for -x in y))`` and not attempt unpacking, but several alternatives were suggested -(or have been suggested) in the past, including: +This proposal maintains that ``f(*x for x in y)`` should be interpreted as +``f((*x for x in y))`` and not attempt further unpacking of the resulting +generator, but several alternatives were suggested (or have been suggested) in +the past, including: * interpreting ``f(*x for x in y)`` as ``f(*(x for x in y)``, * interpreting ``f(*x for x in y)`` as ``f(*(*x for x in y))``, or @@ -602,10 +596,11 @@ x in y))`` and not attempt unpacking, but several alternatives were suggested The reason to prefer this proposal over these alternatives is the preservation of existent conventions for punctuation around generator expressions. Currently, the general rule is that generator expressions must be wrapped in -parentheses except when provided as the sole argument to a function, and we opt -for maintaining that rule even as we allow more kinds of generator expressions. -This option maintains a full symmetry between comprehensions and generator -expressions that use unpacking and those that don't. +parentheses except when provided as the sole argument to a function, and this +proposal suggests maintaining that rule even as we allow more kinds of +generator expressions. This option maintains a full symmetry between +comprehensions and generator expressions that use unpacking and those that +don't. Currently, we have the following conventions:: @@ -633,7 +628,7 @@ make use of unpacking:: Further Generalizing Unpacking Operators ---------------------------------------- -Another suggestion that came out of our discussion involved further +Another suggestion that came out of the discussion involved further generalizing the ``*`` beyond simply allowing it to be used to unpack the expression in a comprehension. Two main flavors of this extension were considered: @@ -656,6 +651,41 @@ understand and to implement) and of only marginal utility, neither is included in this PEP. As such, these forms should continue to raise a ``SyntaxError``, but with a new error message as described above. +Concerns and Disadvantages +========================== + +Although the general consensus from the discussion thread seemed to be that +this syntax was clear and intuitive, several potential downsides and sources +were raised as well. This section aims to summarize those concerns. + +* **Overlap with existing alternatives:** + While the proposed syntax is arguably clearer and more concise, there are + already several ways to accomplish this same thing in Python. + +* **Potential for overuse or abuse:** + Complex uses of unpacking in comprehensions could obscure logic that would be + clearer in an explicit loop or helper function. While this is already a + concern with comprehensions more generally, the addition of ``*`` and ``**`` + may make particularly-complex uses even more difficult to read and + understand. + +* **Function call ambiguity:** + Expressions like ``f(*x for x in y)`` may initially appear ambiguous, as it's + not obvious whether the intent is to unpack the generator or to pass it as a + single argument. Although this proposal retains existing conventions by + treating that form as equivalent to ``f((*x for x in y))``, the distinction + may not be immediately intuitive. + +* **Unclear limitation of scope:** + This proposal restricts unpacking to the top level of the comprehension + expression. These restrictions may seem arbitrary or surprising to users who + expect unpacking to work more generally within expressions. + +* **Effect on External Tools:** + As with any new syntactical structure, making this change would create work + for maintainers of code formatters, linters, and IDEs, to make sure that the + new syntax is supported. + References ========== From ee343ba0f1c44e686303f8df3c97f22385902a11 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Mon, 30 Jun 2025 13:45:53 -0400 Subject: [PATCH 09/27] add example --- peps/pep-9999.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index e6eb1be5aff..6e50b2d091b 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -667,7 +667,9 @@ were raised as well. This section aims to summarize those concerns. clearer in an explicit loop or helper function. While this is already a concern with comprehensions more generally, the addition of ``*`` and ``**`` may make particularly-complex uses even more difficult to read and - understand. + understand. For example, while these situations are likely quite rare, + comprehensions that use unpacking in multiple ways can make it difficult to + know what's being unpacked and when: ``f(*(*x for *x, _ in list_of_lists))``. * **Function call ambiguity:** Expressions like ``f(*x for x in y)`` may initially appear ambiguous, as it's From 201409995c754898472ac263274f9e8748e26f5b Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Mon, 30 Jun 2025 13:57:01 -0400 Subject: [PATCH 10/27] type checkers --- peps/pep-9999.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 6e50b2d091b..6d7465cb9bc 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -685,8 +685,8 @@ were raised as well. This section aims to summarize those concerns. * **Effect on External Tools:** As with any new syntactical structure, making this change would create work - for maintainers of code formatters, linters, and IDEs, to make sure that the - new syntax is supported. + for maintainers of code formatters, linters, type checkers, etc., to make + sure that the new syntax is supported. References ========== From bd610db438fb664de032a5a17add4d26a2f70709 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 3 Jul 2025 20:26:48 -0400 Subject: [PATCH 11/27] add more examples from the standard library --- peps/pep-9999.rst | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 6d7465cb9bc..de2af305d3d 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -470,6 +470,48 @@ the standard library could be rewritten to make use of this new syntax to improve consision and readability. The :ref:`reference` continues to pass all tests with these replacements made. +Replacing Explicit Loops +------------------------ + +Replacing explicit loops compresses multiple lines into one, and avoids the +need for defining and referencing an auxiliary variable. + +* From ``email/_header_value_parser.py``:: + + # current: + comments = [] + for token in self: + comments.extend(token.comments) + return comments + + # improved: + return [*token.comments for token in self] + +* From ``shutil.py``:: + + # current: + ignored_names = [] + for pattern in patterns: + ignored_names.extend(fnmatch.filter(names, pattern)) + return set(ignored_names) + + # improved: + return {*fnmatch.filter(names, pattern) for pattern in patterns} + +* From ``http/cookiejar.py``:: + + # current: + cookies = [] + for domain in self._cookies.keys(): + cookies.extend(self._cookies_for_domain(domain, request)) + return cookies + + # improved: + return [ + *self._cookies_for_domain(domain, request) + for domain in self._cookies.keys() + ] + Replacing from_iterable and Friends ----------------------------------- @@ -558,6 +600,7 @@ referencing an auxiliary variable, reducing clutter. # improved: return self.__class__(*s for s in (self, other)) + Rejected Alternative Proposals ============================== From c521ea8d849dd3b7bac88791a06f22e1cee1e755 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 3 Jul 2025 20:54:13 -0400 Subject: [PATCH 12/27] update date --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index de2af305d3d..4de2816cfcc 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -5,7 +5,7 @@ Sponsor: TBD Status: Draft Type: Standards Track Content-Type: text/x-rst -Created: 30-Jun-2025 +Created: 3-Jul-2025 Python-Version: 3.15 Post-History: `16-Oct-2021 `__, `22-Jun-2025 `__ From 633855176d7ac93c04c8fa5064712941363cedcf Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Fri, 4 Jul 2025 00:47:54 -0400 Subject: [PATCH 13/27] small efficiency note --- peps/pep-9999.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 4de2816cfcc..1021abb3cd1 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -574,7 +574,9 @@ Replacing Double Loops in Comprehensions ---------------------------------------- Replacing double loops in comprehensions avoids the need for defining and -referencing an auxiliary variable, reducing clutter. +referencing an auxiliary variable, reducing clutter. Additionally, using a +single ``LIST_EXTEND`` instead of repeated ``LIST_APPEND`` means that the +updated version tends to be more efficient as well. * From ``multiprocessing.py``:: From 84181e549f39eb95661b7d05ddf8a908e314dd91 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Fri, 4 Jul 2025 17:19:32 -0400 Subject: [PATCH 14/27] remove bytecode examples, add more SO examples, phrasing/spelling --- peps/pep-9999.rst | 162 ++++++++++++++++++++++++---------------------- 1 file changed, 84 insertions(+), 78 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 1021abb3cd1..c80f6a8347f 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,7 +1,7 @@ PEP: 9999 Title: Unpacking in Comprehensions Author: Adam Hartz , Erik Demaine -Sponsor: TBD +Sponsor: Jelle Zijlstra Status: Draft Type: Standards Track Content-Type: text/x-rst @@ -35,8 +35,11 @@ easy to combine a few iterables or dictionaries:: {**dict1, **dict2, **dict3} # dict with the combination of three dicts But if we want to similarly combine an arbitrary number of iterables, we cannot -use unpacking in this same way. Currently, we do have a few options. We could -use explicit looping structures and built in means of combination:: +use unpacking in this same way. + +That said, we do have a few options for combining multiple iterables. We +could, for example, use explicit looping structures and built-in means of +combination:: new_list = [] for it in its: @@ -54,9 +57,7 @@ use explicit looping structures and built in means of combination:: for it in its: yield from it - -Alternatively, we could be more concise by using a comprehension with two -loops:: +Or, we could be more concise by using a comprehension with two loops:: [x for it in its for x in it] {x for it in its for x in it} @@ -76,17 +77,16 @@ Or, for all but the generator, we could use ``functools.reduce``:: functools.reduce(operator.ior, its, (new_set := set())) functools.reduce(operator.ior, its, (new_dict := {})) -As an additional alternative, this PEP proposes extending the unpacking pattern -to enable the use of ``*`` and ``**`` in comprehensions and generator -expressions, for example:: +This PEP proposes allowing unpacking operations to be used in comprehensions as +an additional alternative:: [*it for it in its] # list with the concatenation of iterables in 'its' {*it for it in its} # set with the union of iterables in 'its' {**d for d in dicts} # dict with the combination of dicts in 'dicts' - (*it for it in its) # generator of the concatenation of iterables in 'its' + (*it for it in its) # generator of the concatenation of iterables in 'its' (*it for it in its) This proposal also extends to asynchronous comprehensions and generator -expressions, such that, for example, ``(*x async for x in aits())`` is +expressions, such that, for example, ``(*ait async for ait in aits())`` is equivalent to ``(x async for ait in aits() for x in ait)``. Rationale @@ -95,10 +95,10 @@ Rationale Combining iterable objects together into a single larger object is a common task. One `StackOverflow post `_ -asking about flattening a list of lists, for example, has been viewed 4.6 million -times. Despite this being a common task, the options currently available for -performing this operation concisely require levels of indirection that can make -the resulting code difficult to read and understand. +asking about flattening a list of lists, for example, has been viewed 4.6 +million times. Despite this being a common operation, the options currently +available for performing it concisely require levels of indirection that can +make the resulting code difficult to read and understand. The proposed notation is concise (avoiding the use and repetition of auxiliary variables) and, we expect, intuitive and familiar to programmers familiar with @@ -107,10 +107,15 @@ both comprehensions and unpacking notation. This proposal was motivated in part by a written exam in a Python programming class, where several students used the notation (specifically the ``set`` version) in their solutions, assuming that it already existed in Python. This -suggests that the notation is intuitive, even to those who are learning Python. -By contrast, the existing syntax ``[x for it in its for x in it]`` is one that -students often get wrong, with the natural impulse for many students being to -reverse the order of the ``for`` clauses. +suggests that the notation is intuitive, even to beginners. By contrast, the +existing syntax ``[x for it in its for x in it]`` is one that students often +get wrong, the natural impulse for many students being to reverse the order of +the ``for`` clauses. + +Additionally, though they have not been viewed as many times as the more +general question of flattening lists, there are multiple other instances of +StackOverflow posts asking about the specific syntax proposed here and why it +doesn't work (e.g., [#so1]_, [#so2]_, [#so3]_). See :ref:`examples` for examples of code that could be rewritten more clearly and concisely using the proposed syntax. @@ -124,26 +129,23 @@ Syntax The necessary grammatical changes are allowing the expression in list/set comprehensions and generator expressions to be preceded by a ``*``, and -allowing an alternative form of dictionary comprehension where the expression -is given by a single expression preceded by a ``**`` rather than a ``key: -value`` pair. +allowing an alternative form of dictionary comprehension in which a +double-starred expression can be used in place of a ``key: value`` pair. This can be accomplished by updating the ``listcomp`` and ``setcomp`` rules to use ``star_named_expression`` instead of ``named_expression``:: listcomp[expr_ty]: | '[' a=star_named_expression b=for_if_clauses ']' - | invalid_comprehension setcomp[expr_ty]: | '{' a=star_named_expression b=for_if_clauses '}' - | invalid_comprehension -The rule for ``genexp`` would similarly need to be modified to allow a ``starred_expression``:: +The rule for ``genexp`` would similarly need to be modified to allow a +``starred_expression``:: genexp[expr_ty]: | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')' - | invalid_comprehension The rule for dictionary comprehensions would need to be adjusted as well, to allow for this new form:: @@ -171,21 +173,32 @@ same way as if they were explicitly listed via ``[*expr1, *expr2, ...]``. Similarly, ``{*expr for x in it}`` forms a set union, as if the expressions were explicitly listed via ``{*expr1, *expr2, ...}``; and ``{**expr for x in it}`` combines dictionaries, as if the expressions were explicitly listed via -``{**expr1, **expr2, ...}``, retaining all of the equivalent semantics for -combining collections in this way (e.g., later values replacing earlier values -associated with the same key when combining dictionaries). - -For list and set comprehensions, the generated bytecode between the starred and -unstarred version of the same comprehension should be identical, except for -replacing the opcode for adding a single element to the collection being built -up (``LIST_APPEND`` and ``SET_ADD``, respectively) with the opcode for -combining collections of that type (``LIST_EXTEND`` and ``SET_UPDATE``, -respectively). - -Dictionary comprehensions should follow a similar pattern. The resulting -bytecode will necessarily be somewhat different, but the key difference will be -the use of ``DICT_UPDATE`` instead of ``MAP_ADD`` as the way to add elements to -the new dictionary. +``{**expr1, **expr2, ...}``. These operations should retain all of the +equivalent semantics for combining collections in this way (including, for +example, later values replacing earlier ones in the case of a duplicated key +when combining dictionaries). + +Said another way, the objects created by the following comprehensions:: + + new_list = [*expr for x in its] + new_set = {*expr for x in its} + new_dict = {**expr for d in dicts} + +should be equivalent to the objects created by the following pieces of code, +respectively:: + + new_list = [] + for x in its: + new_list.extend(expr) + + new_set = set() + for x in its: + new_set.update(expr) + + new_dict = {} + for x in dicts: + new_dict.update(expr) + Semantics: Generator Expressions -------------------------------- @@ -199,26 +212,16 @@ generator:: for x in it: yield from expr -For synchronous generator expressions, the generated bytecode for the starred -and unstarred version of the same generator expression should be similar, but -the starred expression should using ``YIELD_FROM`` instead of ``YIELD_VALUE`` -inside the loop. - -For async generator expressions, ``(*expr async for x in ait())``, the -equivalence is more like the following:: +Since ``yield from`` is not allowed inside of async generators, the equivalent +for ``(*expr async for x in ait())``, is more like the following (though of +course this new form should not define or reference the looping variable +``i``):: async def generator(): async for x in ait(): for i in expr: yield i -Since ``YIELD_FROM`` is not allowed inside of async generators, this form -should instead mimic the functionality of the existing ``(z async for x in y -for z in x)`` syntax more directly. The resulting bytecode for, for example, -``(*x for x in y)`` should be the same as the bytecode for ``(z async for x in -y for z in x)``, with the natural exception of the ``STORE_FAST_LOAD_FAST`` -used to bind the variable ``z``. - Interaction with Assignment Expressions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -268,11 +271,10 @@ Error Reporting Currently, the proposed syntax generates a ``SyntaxError`` (via the ``invalid_comprehension`` and ``invalid_dict_comprehension`` rules). -Allowing these forms to be recognized as syntactically valid requires changing -the grammar rule for ``invalid_comprehension`` so that using ``*`` in a -comprehension no longer raises a ``SyntaxError``, as well as removing the rule -for ``invalid_dict_comprehension`` (which currently only checks for ``**`` -being used in a dictionary comprehension). +Allowing these forms to be recognized as syntactically valid requires adjusting +the grammar rules for ``invalid_comprehension`` and +``invalid_dict_comprehension`` to allow the use of ``*`` and ``**``, +respectively. Additional specific error messages should be provided in at least the following cases: @@ -369,7 +371,7 @@ The behavior of all comprehensions that are currently syntactically valid would be unaffected by this change, so we do not anticipate much in the way of backwards-incompatibility concerns. In principle, this change would only affect code that relied on the fact that attempting to use unpacking operations -in comprehensions would raise a ``SyntaxError``, or who relied on the +in comprehensions would raise a ``SyntaxError``, or that relied on the particular phrasing of any of the old error messages being replaced, which we expect to be rare. @@ -388,7 +390,7 @@ is equivalent to the following code:: Taking this approach, we can introduce ``out = [*expr for x in it]`` as instead being equivalent to the following (which uses ``extend`` instead of -``append``:: +``append``):: out = [] for x in it: @@ -467,7 +469,7 @@ Code Examples This section shows some illustrative examples of how small pieces of code from the standard library could be rewritten to make use of this new syntax to -improve consision and readability. The :ref:`reference` continues to pass all +improve concision and readability. The :ref:`reference` continues to pass all tests with these replacements made. Replacing Explicit Loops @@ -574,9 +576,7 @@ Replacing Double Loops in Comprehensions ---------------------------------------- Replacing double loops in comprehensions avoids the need for defining and -referencing an auxiliary variable, reducing clutter. Additionally, using a -single ``LIST_EXTEND`` instead of repeated ``LIST_APPEND`` means that the -updated version tends to be more efficient as well. +referencing an auxiliary variable, reducing clutter and improving performance. * From ``multiprocessing.py``:: @@ -610,11 +610,11 @@ The primary goal when thinking through the specification above was consistency with existing norms around unpacking and comprehensions / generator expressions. One way to interpret this is that the goal was to write the specification so as to require the smallest possible change(s) to the existing -grammar and code generation and letting the existing code inform the surrounding +grammar and code generation, letting the existing code inform the surrounding semantics. -Below we discuss some of the common concerns/alternative proposals that have -come up in dicussions in the past but that are not included in this proposal. +Below we discuss some of the common concerns/alternative proposals that came up +in discussions but that are not included in this proposal. .. _functionargs: @@ -628,7 +628,7 @@ argument to ``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as a reason for not including a similar generalization as part of the proposal. -This proposal maintains that ``f(*x for x in y)`` should be interpreted as +This proposal clarifies that ``f(*x for x in y)`` should be interpreted as ``f((*x for x in y))`` and not attempt further unpacking of the resulting generator, but several alternatives were suggested (or have been suggested) in the past, including: @@ -657,7 +657,7 @@ Currently, we have the following conventions:: f(*{x for x in y}) # pass in elements from the set separately f(*(x for x in y)) # pass in elements from the generator separately (parentheses required) -This proposal opts to maintains those conventions even when the comprehensions +This proposal opts to maintain those conventions even when the comprehensions make use of unpacking:: f([*x for x in y]) # pass in a single list @@ -691,16 +691,17 @@ considered: [*x if isinstance(x, Iterable) else x for x in [[1,2,3], 4]] -Because these variants were deemed to be substantially more complex (both to -understand and to implement) and of only marginal utility, neither is included -in this PEP. As such, these forms should continue to raise a ``SyntaxError``, -but with a new error message as described above. +These variants were considered substantially more complex (both to understand +and to implement) and of only marginal utility, so neither is included in this +PEP. As such, these forms should continue to raise a ``SyntaxError``, but with +a new error message as described above, though it should not be ruled out as a +consideration for future proposals. Concerns and Disadvantages ========================== Although the general consensus from the discussion thread seemed to be that -this syntax was clear and intuitive, several potential downsides and sources +this syntax was clear and intuitive, several concerns and potential downsides were raised as well. This section aims to summarize those concerns. * **Overlap with existing alternatives:** @@ -736,8 +737,13 @@ were raised as well. This section aims to summarize those concerns. References ========== -.. [#guido] Message from Guido van Rossum - (https://mail.python.org/archives/list/python-ideas@python.org/message/CQPULNM6PM623PLXF5Z63BIUZGOSQEKW/) +.. [#guido] `Message from Guido van Rossum `_ + +.. [#so1] `StackOverflow: Unpacking tuples in a python list comprehension (cannot use the *-operator) `_ + +.. [#so2] `StackOverflow: Flatten a nested list using list unpacking in a list comprehension `_ + +.. [#so3] `StackOverflow: Why is it not possible to unpack lists inside a list comprehension? `_ Copyright ========= From 370f61ef111f9170d95549d5e8e410009456dd50 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 12 Jul 2025 23:40:38 -0400 Subject: [PATCH 15/27] small working changes --- peps/pep-9999.rst | 59 ++++++++++++++++++----------------------------- 1 file changed, 22 insertions(+), 37 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index c80f6a8347f..88cc594dd30 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -102,7 +102,9 @@ make the resulting code difficult to read and understand. The proposed notation is concise (avoiding the use and repetition of auxiliary variables) and, we expect, intuitive and familiar to programmers familiar with -both comprehensions and unpacking notation. +both comprehensions and unpacking notation (see :ref:`examples` for examples of +code from the standard library that could be rewritten more clearly and +concisely using the proposed syntax). This proposal was motivated in part by a written exam in a Python programming class, where several students used the notation (specifically the ``set`` @@ -112,14 +114,6 @@ existing syntax ``[x for it in its for x in it]`` is one that students often get wrong, the natural impulse for many students being to reverse the order of the ``for`` clauses. -Additionally, though they have not been viewed as many times as the more -general question of flattening lists, there are multiple other instances of -StackOverflow posts asking about the specific syntax proposed here and why it -doesn't work (e.g., [#so1]_, [#so2]_, [#so3]_). - -See :ref:`examples` for examples of code that could be rewritten more clearly -and concisely using the proposed syntax. - Specification ============= @@ -268,13 +262,10 @@ the containing scope) will be modified at those points in time:: Error Reporting --------------- -Currently, the proposed syntax generates a ``SyntaxError`` (via the -``invalid_comprehension`` and ``invalid_dict_comprehension`` rules). - -Allowing these forms to be recognized as syntactically valid requires adjusting -the grammar rules for ``invalid_comprehension`` and -``invalid_dict_comprehension`` to allow the use of ``*`` and ``**``, -respectively. +Currently, the proposed syntax generates a ``SyntaxError``. Allowing these +forms to be recognized as syntactically valid requires adjusting the grammar +rules for ``invalid_comprehension`` and ``invalid_dict_comprehension`` to allow +the use of ``*`` and ``**``, respectively. Additional specific error messages should be provided in at least the following cases: @@ -628,10 +619,10 @@ argument to ``f(*x for x in y)``. In the original :pep:`448`, this ambiguity was cited as a reason for not including a similar generalization as part of the proposal. -This proposal clarifies that ``f(*x for x in y)`` should be interpreted as -``f((*x for x in y))`` and not attempt further unpacking of the resulting -generator, but several alternatives were suggested (or have been suggested) in -the past, including: +This proposal suggests that ``f(*x for x in y)`` should be interpreted as +``f((*x for x in y))`` and should not attempt further unpacking of the +resulting generator, but several alternatives were suggested in our discussion +(and/or have been suggested in the past), including: * interpreting ``f(*x for x in y)`` as ``f(*(x for x in y)``, * interpreting ``f(*x for x in y)`` as ``f(*(*x for x in y))``, or @@ -708,21 +699,21 @@ were raised as well. This section aims to summarize those concerns. While the proposed syntax is arguably clearer and more concise, there are already several ways to accomplish this same thing in Python. -* **Potential for overuse or abuse:** - Complex uses of unpacking in comprehensions could obscure logic that would be - clearer in an explicit loop or helper function. While this is already a - concern with comprehensions more generally, the addition of ``*`` and ``**`` - may make particularly-complex uses even more difficult to read and - understand. For example, while these situations are likely quite rare, - comprehensions that use unpacking in multiple ways can make it difficult to - know what's being unpacked and when: ``f(*(*x for *x, _ in list_of_lists))``. - * **Function call ambiguity:** Expressions like ``f(*x for x in y)`` may initially appear ambiguous, as it's not obvious whether the intent is to unpack the generator or to pass it as a single argument. Although this proposal retains existing conventions by - treating that form as equivalent to ``f((*x for x in y))``, the distinction - may not be immediately intuitive. + treating that form as equivalent to ``f((*x for x in y))``, that may not be + immediately obvious. + +* **Potential for overuse or abuse:** + Complex uses of unpacking in comprehensions could obscure logic that would be + clearer in an explicit loop. While this is already a concern with + comprehensions more generally, the addition of ``*`` and ``**`` may make + particularly-complex uses even more difficult to read and understand at a + glance. For example, while these situations are likely rare, comprehensions + that use unpacking in multiple ways can make it difficult to know what's + being unpacked and when: ``f(*(*x for *x, _ in list_of_lists))``. * **Unclear limitation of scope:** This proposal restricts unpacking to the top level of the comprehension @@ -739,12 +730,6 @@ References .. [#guido] `Message from Guido van Rossum `_ -.. [#so1] `StackOverflow: Unpacking tuples in a python list comprehension (cannot use the *-operator) `_ - -.. [#so2] `StackOverflow: Flatten a nested list using list unpacking in a list comprehension `_ - -.. [#so3] `StackOverflow: Why is it not possible to unpack lists inside a list comprehension? `_ - Copyright ========= From 3ea97813aae6b0a393c688a252216249514c96ed Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 13 Jul 2025 23:47:19 -0400 Subject: [PATCH 16/27] add brief discussion of other languages --- peps/pep-9999.rst | 50 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 88cc594dd30..0aee9b8a3de 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -725,6 +725,56 @@ were raised as well. This section aims to summarize those concerns. for maintainers of code formatters, linters, type checkers, etc., to make sure that the new syntax is supported. +Other Languages +=============== + +Quite a few other languages support this kind of flattening with syntax similar +to what is already available in Python, but support for using unpacking syntax +within comprehensions is rare. This section provides a brief summary of +support for similar syntax in a few other languages. + +Many languages that support comprehensions for support double loops:: + + # python + [x for xs in [[1,2,3], [], [4,5]] for x in xs * 2] + + -- haskell + [x | xs <- [[1,2,3], [], [4,5]], x <- xs ++ xs] + + # julia + [x for xs in [[1,2,3], [], [4,5]] for x in [xs; xs]] + + ; clojure + (for [xs [[1 2 3] [] [4 5]] x (concat xs xs)] x) + +Several other languages (even those without comprehensions) support these +operations via a built-in function/method to support flattening of nested +structures:: + + # python + list(itertools.chain(*(xs*2 for xs in [[1,2,3], [], [4,5]]))) + + // Javascript + [[1,2,3], [], [4,5]].flatMap(xs => [...xs, ...xs]) + + -- haskell + concat (map (\x -> x ++ x) [[1,2,3], [], [4,5]]) + + # ruby + [[1, 2, 3], [], [4, 5]].flat_map {|e| e * 2} + +However, languages that support both comprehension and unpacking do not tend to +allow unpacking within a comprehension. For example, the following expression +in Julia currently leads to a syntax error:: + + [xs... for xs in [[1,2,3], [], [4,5]]] + +As one counterexample, support for a similar syntax was recently added to `Civet +`_. For example, the following is a valid comprehension in +Civet, making ue of Javascript's ``...`` syntax for unpacking:: + + for xs of [[1,2,3], [], [4,5]] then ...(xs++xs) + References ========== From 0ab2ccaeac0bd9155ea480f4960ee24c4d5e8d88 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 13 Jul 2025 23:55:01 -0400 Subject: [PATCH 17/27] reorder sections --- peps/pep-9999.rst | 172 +++++++++++++++++++++++----------------------- 1 file changed, 86 insertions(+), 86 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 0aee9b8a3de..cc1966d41ff 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -367,92 +367,6 @@ particular phrasing of any of the old error messages being replaced, which we expect to be rare. -How to Teach This -================= - -Currently, a common way to introduce the notion of comprehensions (which is -employed by the Python Tutorial) is to demonstrate equivalent code. For -example, this method would say that, for example, ``out = [expr for x in it]`` -is equivalent to the following code:: - - out = [] - for x in it: - out.append(expr) - -Taking this approach, we can introduce ``out = [*expr for x in it]`` as instead -being equivalent to the following (which uses ``extend`` instead of -``append``):: - - out = [] - for x in it: - out.extend(expr) - -Set and dict comprehensions that make use of unpacking can also be introduced -by a similar analogy:: - - # equivalent to out = {expr for x in it} - out = set() - for x in it: - out.add(expr) - - # equivalent to out = {*expr for x in it} - out = set() - for x in it: - out.update(expr) - - # equivalent to out = {k_expr: v_expr for x in it} - out = {} - for x in it: - out[k_expr] = v_expr - - # equivalent to out = {**expr for x in it} - out = {} - for x in it: - out.update(expr) - -And we can take a similar approach to illustrate the behavior of generator -expressions that involve unpacking:: - - # equivalent to g = (expr for x in it) - def generator(): - for x in it: - yield expr - g = generator() - - # equivalent to g = (*expr for x in it) - def generator(): - for x in it: - yield from expr - g = generator() - -We can then generalize from these specific examples to the idea that, -wherever a non-starred comprehension/genexp would use an operator that -adds a single element to a collection, the starred would instead use -an operator that adds multiple elements to that collection. - -Alternatively, we don't need to think of the two ideas as separate; instead, -with the new syntax, we can think of ``out = [...x... for x in it]`` as -equivalent to the following code [#guido]_, regardless of whether or not -``...x...`` uses ``*``:: - - out = [] - for x in it: - out.extend([...x...]) - -Similarly, we can think of ``out = {...x... for x in it}`` as equivalent to the -following code, regardless of whether or not ``...x...`` uses ``*`` or ``**`` -or ``:``:: - - out = set() - for x in it: - out.update({...x...}) - -These examples are equivalent in the sense that the output they produce would -be the same in both the version with the comprehension and the version without -it, but note that the non-comprehension version is slightly less efficient due -to making new lists/sets/dictionaries before each ``extend`` or ``update``, -which is unnecessary in the version that uses comprehensions. - .. _examples: Code Examples @@ -594,6 +508,92 @@ referencing an auxiliary variable, reducing clutter and improving performance. return self.__class__(*s for s in (self, other)) +How to Teach This +================= + +Currently, a common way to introduce the notion of comprehensions (which is +employed by the Python Tutorial) is to demonstrate equivalent code. For +example, this method would say that, for example, ``out = [expr for x in it]`` +is equivalent to the following code:: + + out = [] + for x in it: + out.append(expr) + +Taking this approach, we can introduce ``out = [*expr for x in it]`` as instead +being equivalent to the following (which uses ``extend`` instead of +``append``):: + + out = [] + for x in it: + out.extend(expr) + +Set and dict comprehensions that make use of unpacking can also be introduced +by a similar analogy:: + + # equivalent to out = {expr for x in it} + out = set() + for x in it: + out.add(expr) + + # equivalent to out = {*expr for x in it} + out = set() + for x in it: + out.update(expr) + + # equivalent to out = {k_expr: v_expr for x in it} + out = {} + for x in it: + out[k_expr] = v_expr + + # equivalent to out = {**expr for x in it} + out = {} + for x in it: + out.update(expr) + +And we can take a similar approach to illustrate the behavior of generator +expressions that involve unpacking:: + + # equivalent to g = (expr for x in it) + def generator(): + for x in it: + yield expr + g = generator() + + # equivalent to g = (*expr for x in it) + def generator(): + for x in it: + yield from expr + g = generator() + +We can then generalize from these specific examples to the idea that, +wherever a non-starred comprehension/genexp would use an operator that +adds a single element to a collection, the starred would instead use +an operator that adds multiple elements to that collection. + +Alternatively, we don't need to think of the two ideas as separate; instead, +with the new syntax, we can think of ``out = [...x... for x in it]`` as +equivalent to the following code [#guido]_, regardless of whether or not +``...x...`` uses ``*``:: + + out = [] + for x in it: + out.extend([...x...]) + +Similarly, we can think of ``out = {...x... for x in it}`` as equivalent to the +following code, regardless of whether or not ``...x...`` uses ``*`` or ``**`` +or ``:``:: + + out = set() + for x in it: + out.update({...x...}) + +These examples are equivalent in the sense that the output they produce would +be the same in both the version with the comprehension and the version without +it, but note that the non-comprehension version is slightly less efficient due +to making new lists/sets/dictionaries before each ``extend`` or ``update``, +which is unnecessary in the version that uses comprehensions. + Rejected Alternative Proposals ============================== From c7306b69b7566acb566c14012c6c23d852bd59b7 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 19:55:51 -0400 Subject: [PATCH 18/27] move to pep 798 --- peps/{pep-9999.rst => pep-0798.rst} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename peps/{pep-9999.rst => pep-0798.rst} (99%) diff --git a/peps/pep-9999.rst b/peps/pep-0798.rst similarity index 99% rename from peps/pep-9999.rst rename to peps/pep-0798.rst index cc1966d41ff..f8ae78f1396 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-0798.rst @@ -1,11 +1,11 @@ -PEP: 9999 +PEP: 798 Title: Unpacking in Comprehensions Author: Adam Hartz , Erik Demaine Sponsor: Jelle Zijlstra +Discussions-To: Pending Status: Draft Type: Standards Track -Content-Type: text/x-rst -Created: 3-Jul-2025 +Created: 19-Jul-2025 Python-Version: 3.15 Post-History: `16-Oct-2021 `__, `22-Jun-2025 `__ From c6cb42bf05e567d50abbef8dd8a6a24b300867d4 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 20:09:21 -0400 Subject: [PATCH 19/27] fix a few typos --- peps/pep-0798.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index f8ae78f1396..2d3fa794a13 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -53,7 +53,7 @@ combination:: for d in dicts: new_dict.update(d) - def generator(): + def new_generator(): for it in its: yield from it @@ -733,7 +733,7 @@ to what is already available in Python, but support for using unpacking syntax within comprehensions is rare. This section provides a brief summary of support for similar syntax in a few other languages. -Many languages that support comprehensions for support double loops:: +Many languages that support comprehensions support double loops:: # python [x for xs in [[1,2,3], [], [4,5]] for x in xs * 2] @@ -771,7 +771,7 @@ in Julia currently leads to a syntax error:: As one counterexample, support for a similar syntax was recently added to `Civet `_. For example, the following is a valid comprehension in -Civet, making ue of Javascript's ``...`` syntax for unpacking:: +Civet, making use of Javascript's ``...`` syntax for unpacking:: for xs of [[1,2,3], [], [4,5]] then ...(xs++xs) From 73429c0d963b0ee7e2748e66d6880f8ce7b608da Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 20:12:47 -0400 Subject: [PATCH 20/27] add Jelle to codeowners --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e000d3934b1..53ddbd0b929 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -672,6 +672,7 @@ peps/pep-0791.rst @vstinner peps/pep-0792.rst @dstufft peps/pep-0793.rst @encukou peps/pep-0794.rst @brettcannon +peps/pep-0798.rst @JelleZijlstra # ... peps/pep-0801.rst @warsaw # ... From f7171dbb21462ddaa5de6d4e551ad62277f4ba5f Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 20:30:35 -0400 Subject: [PATCH 21/27] code formatting for grammar blocks --- peps/pep-0798.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 2d3fa794a13..b896d7a7841 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -127,7 +127,9 @@ allowing an alternative form of dictionary comprehension in which a double-starred expression can be used in place of a ``key: value`` pair. This can be accomplished by updating the ``listcomp`` and ``setcomp`` rules to -use ``star_named_expression`` instead of ``named_expression``:: +use ``star_named_expression`` instead of ``named_expression``: + +.. code:: text listcomp[expr_ty]: | '[' a=star_named_expression b=for_if_clauses ']' @@ -136,13 +138,17 @@ use ``star_named_expression`` instead of ``named_expression``:: | '{' a=star_named_expression b=for_if_clauses '}' The rule for ``genexp`` would similarly need to be modified to allow a -``starred_expression``:: +``starred_expression``: + +.. code:: text genexp[expr_ty]: | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')' The rule for dictionary comprehensions would need to be adjusted as well, to -allow for this new form:: +allow for this new form: + +.. code:: text dictcomp[expr_ty]: | '{' a=double_starred_kvpair b=for_if_clauses '}' From bf991890523bd40c5230960075e8d8eac5fbbbe1 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 20:39:16 -0400 Subject: [PATCH 22/27] wording of concerns/disadvantages --- peps/pep-0798.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index b896d7a7841..ebfc374ccb4 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -709,8 +709,8 @@ were raised as well. This section aims to summarize those concerns. Expressions like ``f(*x for x in y)`` may initially appear ambiguous, as it's not obvious whether the intent is to unpack the generator or to pass it as a single argument. Although this proposal retains existing conventions by - treating that form as equivalent to ``f((*x for x in y))``, that may not be - immediately obvious. + treating that form as equivalent to ``f((*x for x in y))``, that equivalence + may not be immediately obvious. * **Potential for overuse or abuse:** Complex uses of unpacking in comprehensions could obscure logic that would be @@ -723,11 +723,11 @@ were raised as well. This section aims to summarize those concerns. * **Unclear limitation of scope:** This proposal restricts unpacking to the top level of the comprehension - expression. These restrictions may seem arbitrary or surprising to users who - expect unpacking to work more generally within expressions. + expression, but some users may expect that the unpacking operator is being + further generalized as discussed in :ref:`moregeneral`. * **Effect on External Tools:** - As with any new syntactical structure, making this change would create work + As with any change to Python's syntax, making this change would create work for maintainers of code formatters, linters, type checkers, etc., to make sure that the new syntax is supported. From 413017e4f4f3cb3146276a5c5b455084a2f0ec05 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 21:15:20 -0400 Subject: [PATCH 23/27] fix filenames in examples section --- peps/pep-0798.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index ebfc374ccb4..b77be7f2cb4 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -489,7 +489,7 @@ Replacing Double Loops in Comprehensions Replacing double loops in comprehensions avoids the need for defining and referencing an auxiliary variable, reducing clutter and improving performance. -* From ``multiprocessing.py``:: +* From ``importlib/resources/readers.py``:: # current: children = (child for path in self._paths for child in path.iterdir()) @@ -497,7 +497,7 @@ referencing an auxiliary variable, reducing clutter and improving performance. # improved: children = (*path.iterdir() for path in self._paths) -* From ``Lib/asyncio/base_events.py``:: +* From ``asyncio/base_events.py``:: # current: exceptions = [exc for sub in exceptions for exc in sub] From 47a28f6cb33955b07ebcc7847f1d3a5c84d88b65 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 21:27:51 -0400 Subject: [PATCH 24/27] add prefixes to reference names --- peps/pep-0798.rst | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index b77be7f2cb4..4d640b7c559 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -102,9 +102,9 @@ make the resulting code difficult to read and understand. The proposed notation is concise (avoiding the use and repetition of auxiliary variables) and, we expect, intuitive and familiar to programmers familiar with -both comprehensions and unpacking notation (see :ref:`examples` for examples of -code from the standard library that could be rewritten more clearly and -concisely using the proposed syntax). +both comprehensions and unpacking notation (see :ref:`pep798-examples` for +examples of code from the standard library that could be rewritten more clearly +and concisely using the proposed syntax). This proposal was motivated in part by a written exam in a Python programming class, where several students used the notation (specifically the ``set`` @@ -157,11 +157,11 @@ No change should be made to the way that argument unpacking is handled in function calls, i.e., the general rule that generator expressions provided as the sole argument to functions do not require additional redundant parentheses should be retained. Note that this implies that, for example, ``f(*x for x in -it)`` is equivalent to ``f((*x for x in it))`` (see :ref:`functionargs` for -more discussion). +it)`` is equivalent to ``f((*x for x in it))`` (see :ref:`pep798-functionargs` +for more discussion). ``*`` and ``**`` should only be allowed at the top-most level of the expression -in the comprehension (see :ref:`moregeneral` for more discussion). +in the comprehension (see :ref:`pep798-moregeneral` for more discussion). Semantics: List/Set/Dict Comprehensions @@ -325,7 +325,7 @@ cases: * The phrasing of some other existing error messages should similarly be adjusted to account for the presence of the new syntax, and/or to clarify ambiguous or confusing cases relating to unpacking more generally - (particularly those mentioned in :ref:`moregeneral`), for example:: + (particularly those mentioned in :ref:`pep798-moregeneral`), for example:: >>> [*x if x else y] File "", line 1 @@ -352,7 +352,7 @@ cases: SyntaxError: cannot use dict unpacking on only part of a conditional expression -.. _reference: +.. _pep798-reference: Reference Implementation ======================== @@ -373,15 +373,15 @@ particular phrasing of any of the old error messages being replaced, which we expect to be rare. -.. _examples: +.. _pep798-examples: Code Examples ============= This section shows some illustrative examples of how small pieces of code from the standard library could be rewritten to make use of this new syntax to -improve concision and readability. The :ref:`reference` continues to pass all -tests with these replacements made. +improve concision and readability. The :ref:`pep798-reference` continues to +pass all tests with these replacements made. Replacing Explicit Loops ------------------------ @@ -579,7 +579,7 @@ an operator that adds multiple elements to that collection. Alternatively, we don't need to think of the two ideas as separate; instead, with the new syntax, we can think of ``out = [...x... for x in it]`` as -equivalent to the following code [#guido]_, regardless of whether or not +equivalent to the following code [#pep798-guido]_, regardless of whether or not ``...x...`` uses ``*``:: out = [] @@ -613,7 +613,7 @@ semantics. Below we discuss some of the common concerns/alternative proposals that came up in discussions but that are not included in this proposal. -.. _functionargs: +.. _pep798-functionargs: Starred Generators as Function Arguments ---------------------------------------- @@ -665,7 +665,7 @@ make use of unpacking:: f(*{*x for x in y}) # pass in elements from the set separately f(*(*x for x in y)) # pass in elements from the generator separately (parentheses required) -.. _moregeneral: +.. _pep798-moregeneral: Further Generalizing Unpacking Operators ---------------------------------------- @@ -724,7 +724,7 @@ were raised as well. This section aims to summarize those concerns. * **Unclear limitation of scope:** This proposal restricts unpacking to the top level of the comprehension expression, but some users may expect that the unpacking operator is being - further generalized as discussed in :ref:`moregeneral`. + further generalized as discussed in :ref:`pep798-moregeneral`. * **Effect on External Tools:** As with any change to Python's syntax, making this change would create work @@ -784,7 +784,7 @@ Civet, making use of Javascript's ``...`` syntax for unpacking:: References ========== -.. [#guido] `Message from Guido van Rossum `_ +.. [#pep798-guido] `Message from Guido van Rossum `_ Copyright ========= From 561600083de9c8dad71a49da619c41560f7dbcf9 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 21:34:09 -0400 Subject: [PATCH 25/27] add reference to itertools.chain.from_iterable to the motivation section --- peps/pep-0798.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 4d640b7c559..e75fd67e03e 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -64,13 +64,18 @@ Or, we could be more concise by using a comprehension with two loops:: {key: value for d in dicts for key, value in d.items()} (x for it in its for x in it) -Or, we could use ``itertools.chain``:: +Or, we could use ``itertools.chain`` or ``itertools.chain.from_iterable``:: list(itertools.chain(*its)) set(itertools.chain(*its)) dict(itertools.chain(*(d.items() for d in dicts))) itertools.chain(*its) + list(itertools.chain.from_iterable(its)) + set(itertools.chain.from_iterable(its)) + dict(itertools.chain.from_iterable(d.items() for d in dicts)) + itertools.chain.from_iterable(its) + Or, for all but the generator, we could use ``functools.reduce``:: functools.reduce(operator.iconcat, its, (new_list := [])) From e3434cf9c2453731cbd0863cc2e5aa1e51910bd1 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 21:38:32 -0400 Subject: [PATCH 26/27] remove reference to performance improvement --- peps/pep-0798.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index e75fd67e03e..8bb0340a7ce 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -492,7 +492,7 @@ Replacing Double Loops in Comprehensions ---------------------------------------- Replacing double loops in comprehensions avoids the need for defining and -referencing an auxiliary variable, reducing clutter and improving performance. +referencing an auxiliary variable, reducing clutter. * From ``importlib/resources/readers.py``:: From d7f77c1c8503650cc64e8e6a8ea386292a82a858 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 19 Jul 2025 21:55:26 -0400 Subject: [PATCH 27/27] change capitalization of suggested error messages --- peps/pep-0798.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 8bb0340a7ce..25649fde238 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -336,13 +336,13 @@ cases: File "", line 1 [*x if x else y] ^^^^^^^^^^^^^^ - SyntaxError: invalid starred expression. did you forget to wrap the conditional expression in parentheses? + SyntaxError: invalid starred expression. Did you forget to wrap the conditional expression in parentheses? >>> {**x if x else y} File "", line 1 {**x if x else y} ^^^^^^^^^^^^^^^ - SyntaxError: invalid double starred expression. did you forget to wrap the conditional expression in parentheses? + SyntaxError: invalid double starred expression. Did you forget to wrap the conditional expression in parentheses? >>> [x if x else *y] File "", line 1