Skip to content

gh-143055: Delegate to subiterators when unpacking in generator expressions#152550

Draft
graingert wants to merge 3 commits into
python:mainfrom
graingert:pep798-genexp-delegation
Draft

gh-143055: Delegate to subiterators when unpacking in generator expressions#152550
graingert wants to merge 3 commits into
python:mainfrom
graingert:pep798-genexp-delegation

Conversation

@graingert

@graingert graingert commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Unpacking a sub-iterable with * in a generator expression (PEP 798) now delegates to the sub-iterable using yield from semantics, rather than re-yielding each value with a manual loop.

Sync generator expressions

(*sub for sub in subs) now forwards send() and throw() to the sub-iterator currently being unpacked, and discards the sub-iterator's return value:

>>> def sub():
...     while True:
...         print("sub got", (yield "value"))
>>> g = (*sub() for _ in range(1))
>>> next(g)
'value'
>>> g.send(42)
sub got 42
'value'

Asynchronous generator expressions

* unpacking is synchronous (per PEP 448), so the sub-iterable is always a sync iterable. But it is delegated to from inside an async generator, where each yielded value must be wrapped as an async-generator value — otherwise the async-generator machinery treats it as an awaited value. The throw path is the subtle case: _gen_throw forwards throws to the sub in the runtime and returns the result bypassing any bytecode-level wrap (and that same path is shared with await/async for passthrough, which must stay unwrapped).

To handle this, a small internal iterator _PyAsyncGenUnpack wraps the sync iterable so that every value it produces — via __next__, am_send, and throw — is wrapped, while the return value and exceptions pass through unwrapped. It is constructed via a new INTRINSIC_ASYNC_GEN_UNPACK intrinsic emitted only in the async-generator case. As a result asend(), athrow() and aclose() are forwarded to the sub-iterator's send(), throw() and close().

No async yield from / PEP 828 syntax is introduced; this is purely about delegating PEP 798 unpacking.

Notes

  • The PYC_MAGIC_NUMBER and the managed static type count are bumped (new helper type + changed codegen).
  • This makes 3.16 generator-expression unpacking semantically differ from 3.15 (which shipped the re-yield behaviour). It may be worth considering for 3.15 as well.

Tests

New unittest.TestCase classes in Lib/test/test_unpack_ex.py cover sync send/throw/close forwarding and async asend/athrow/aclose forwarding. The async tests drive the generators by hand (no IsolatedAsyncioTestCase/asyncio), matching the convention in test_asyncgen.py.

🤖 Generated with Claude Code

… expressions

Unpacking a sub-iterable with `*` in a generator expression (PEP 798) now
delegates to the sub-iterable using `yield from` semantics, so that values
sent with send() and exceptions thrown with throw() are forwarded to the
sub-iterator (and the sub-iterator's return value is discarded).

This also works in asynchronous generator expressions.  Since `*` unpacking
is synchronous, the sub-iterable is a sync iterable, but it is delegated to
from inside an async generator, so each produced value must be wrapped as an
async-generator value -- including values produced in response to asend() and
athrow().  A new internal `_PyAsyncGenUnpack` iterator (constructed via the
new INTRINSIC_ASYNC_GEN_UNPACK intrinsic) wraps the sync iterable so that
asend(), athrow() and aclose() are forwarded to the sub-iterator's send(),
throw() and close().

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@read-the-docs-community

read-the-docs-community Bot commented Jun 29, 2026

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #33350739 | 📁 Comparing 2fdf024 against main (2670cb0)

  🔍 Preview build  

3 files changed
± reference/expressions.html
± whatsnew/3.16.html
± whatsnew/changelog.html

graingert and others added 2 commits June 29, 2026 08:44
- async_gen_unpack_throw: validate arity with _PyArg_CheckPositional so that
  calling throw() with no arguments on the internal wrapper (reachable via
  ag_await while suspended at the delegation) raises TypeError instead of
  crashing the interpreter via gen_set_exception(NULL, ...).
- Correct the whatsnew/NEWS/reference wording: an async generator
  expression's aclose() throws GeneratorExit into the sub-iterator via its
  throw(), rather than calling close().
- Add a regression test for the throw() arity check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nalyzer

Add the new static type to Tools/c-analyzer/cpython/globals-to-fix.tsv
alongside the other genobject static types, so the check-c-globals CI step
passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant