gh-137026: Add an explainer guide for asyncio

[anordin95]

Explainer guide for asyncio

gh-137026: HOWTO article for asyncio, and a reference to it from the main page of the asyncio docs.

---

Hi!

I've used Python's asyncio a couple times now, but never really felt confident in my mental model of how it fundamentally works and therefore how I can best leverage it. The official docs provide good documentation for each specific function in the package, but, in my opinion, are missing a cohesive overview of the systems design and architecture. Something that could help the user understand the why and how behind the recommended patterns. And a way to help the user make informed decisions about which tool in the asyncio toolkit they ought to grab, or to recognize when asyncio is the entirely wrong toolkit. I thought I'd take a stab at filling that gap and contributing back to a community that's given so much!

---

📚 Documentation preview 📚: https://cpython-previews--137215.org.readthedocs.build/

[anordin95]

Personally, I’m not sure why this design decision was made and find it rather confuses the meaning of await: asynchronously wait. If someone here does know, please tell me so I can update this section appropriately! I think providing such context can aid significantly with peoples understanding (myself included!).

For reference, I'd imagine either having coroutines yield in their __await__ or disallowing await coroutine entirely (thereby effectively mandating people who want to use coroutines must use coroutine.send()).

[StanFromIreland]

I will review further once the lines are wrapped, as now it will invalidate suggestions.

[StanFromIreland]

Suggested change

:Author: Alexander Nordin

Please see our style guide, new documentation should not have such text, as it discourages others from contributing.

[StanFromIreland]

This comment has yet to be addressed.

[ZeroIntensity]

Hm, maybe we should make an exception to this rule here. This isn't getting a whatsnew or blurb entry, so @anordin95 has no other way to be acknowledged for his efforts.

The devguide doesn't do a great job of explaining why author attribution discourages other people from contributing; would you mind pointing me to the issue where this was discussed? I'm curious if this was based on actual contributor feedback or just speculation.

[StanFromIreland]

This was decided by the EB IIRC. It has been done for the latest HOWTO (which I think is the only one since the guide’s introduction).

[ZeroIntensity]

Is there a link to the EB discussion/decision? I'm not really familiar with how their governance process works.

[picnixz]

A NEWS entry can be added under the documentation category.

[anordin95]

I had noticed most of the other HOWTO's had a byline and figured this would fit the pattern. And given, the nature of HOWTO's (long form articles imbued with some style, humor and flourish from the author) as opposed to other parts of the documentation, thought it made sense. The byline also provides context for the personal references (like "That might sound odd to you. Frankly, it was to me too.").

Granted, I also imagine my own vanity or desire for recognition is playing into this a decent bit.

[StanFromIreland]

Such personalization is discouraged, see this issue

[anordin95]

I generally agree that too much personalization or flourish can be distracting, but I think there's a balance to be struck. Sometimes, it can be of great use to the reader. Furthermore, of any place in the docs, I think explainers/HOWTOs are most suited for this kind of writing.

In the issue you referenced, someone also replied to this effect: #62480 (comment)

[ZeroIntensity]

I agree that "you" adds a nice personal touch to tutorials, but I'll leave it up to the documentation experts.

[StanFromIreland]

I was not talking about the „you”, I was instead referring to this:

During my own asyncio learning process, a few aspects particually drove my
curiosity (read: drove me nuts).

Even if for some odd reason we do want to keep this, we should not have an idiom which will be problematic for translators and (as the devguide states, use simple language) readers.

[anordin95]

I assume you want to remove the whole sentence, yes? The "odd reason" I think it should be kept is that it adds useful context and relates to the the user; getting confused and even frustrated by asyncio is okay and happened for the author of the article too. A bit of personality can likely help folks feel less intimidated by confusing subjects. And I feel it's a much more natural lead in to the list of learning objectives that follows.

Similarly I think something is lost when removing a bit of humor -- "(read: drove me nuts)", but I hear you on translation difficulties. I could replace it with something more straightforward like: "(read: confused the heck out of me)".

[ZeroIntensity]

I'm going to commit to reviewing this if nobody else has the time. I think is generally an improvement to the documentation, but definitely needs some work still. Thanks for working on this!

[ZeroIntensity]

Ok, well, not only one. There can be multiple, just coroutines are only bound to one.

[anordin95]

Mmm agreed! Though, do you think it's necessary to immediately make that clarification? I'm concerned it'll add detail for newbies (that's not crucially important at the moment) and advanced readers could likely figure that out on their own.

[ZeroIntensity]

Sure, I'm fine with not explaining it here. But, I don't think we should lie and say that there's "only one" in the introduction.

[anordin95]

Mmm ok fair enough. I've removed the "and there's only one" part.

[ZeroIntensity]

Please add a distinction between native coroutines and other coroutines. async def functions return native coroutines (types.CoroutineType), but some things (such as C code) will return an object that simply implements the coroutine protocol (collections.abc.Coroutine).

Almost all coroutines are native coroutines, but people generally only need something that implements the coroutine protocol (not the additional frame shenanigans that types.CoroutineType provides).

[anordin95]

Hm, I'm a bit hesitant to add this distinction unless it serves an important and soon to be useful learning goal. Folks are already having to juggle a number of new similar-sounding concepts when going through this article and I'd prefer not to add to their plate if possible. What was the reasoning you had in mind for why this should be included early on?

[ZeroIntensity]

This is coming off frustration with my own library that exposes coroutines to the C API. Unfortunately, you can't make a native coroutine from the (public) C API, so you have to implement a type that implements collections.abc.Coroutine. That means when people use inspect.iscoroutine or similar to check if something can be used with await, it breaks with C API coroutines, because those check for native coroutines instead of the coroutine protocol. I want to encourage people to use the coroutine protocol instead of native coroutines for support there, and documenting the distinction will help with that.

[anordin95]

Ah that does sound frustrating! Thank you for the context.

To ensure I understand, you want to inform folks of the difference between native coroutines & coroutines that abide by the abc.Coroutine spec; that way, they don't misuse inspect.iscoroutine?

I'm still wary of adding that distinction in this section. Regular coroutines have just been introduced. Commenting to warn them of a specific, sharp-corner that they might run in to a fair bit further down the line using other libraries doesn't feel like it belongs. I feel that's more of an advanced user edge-case. Perhaps a note, like the one seen below (from abc.Coroutine) could be modified slightly and added to inspect.iscoroutine to prevent others from encountering this tricksome situation?

[ZeroIntensity]

A note in inspect.iscoroutine/types.CoroutineType is fine, but we should link to one of those here to give something clickable for users.

[anordin95]

Can you draft an example of what you mean? I can't really do so without it feeling clunky and misplaced. Here's what I had in mind:

"Be aware, native coroutines (those made via async def) and coroutines which implement abc.Coroutine but are produced via C functions differ in an important way: inspect.iscoroutine() would answer False for the latter."

But I don't like it. I really don't think this discussion belongs in the intro to coroutines section. It's too specific and somewhat advanced. I fear it'll just confuse folks.

I do also think we're talking in circles a bit here, so it might be useful to have someone else weigh in?

[ZeroIntensity]

There should be a mention somewhere here that in CPython, a (native) coroutine is exactly the same as a generator.

[anordin95]

Mmmm yes. That will provide some clarity as to the currently odd/surprising appearance of the StopIteration exception too. I'll figure out a way to work that in.

[anordin95]

I originally tried to work it in here, but it felt a bit odd. It felt more natural in the coroutines section, so I put it there. Give it a look and lemme know!

[StanFromIreland]

There are still many minor infringements of the Style guide (e.g. simple English) which could be addressed in one pass.

You can also use Sphinx cross references when discussing specific keywords, functions etc.

[StanFromIreland]

Suggested change

	During my own asyncio learning process, a few aspects particually drove my
	curiosity (read: drove me nuts).

I think this is unnecessary, as I've said before on a review comment that is now disjointed after the lines were wrapped. The idiom in particular will be confusing to translators and people new to English.

[StanFromIreland]

This comment has yet to be addressed.

[StanFromIreland]

You can add a codeblock in a title.

[anordin95]

Yes I am aware. I purposely avoiding doing so for this and the await section titles. I think in these cases the usage is pretty clear from context. I did so because inlining these as code resulted in them being bolded in the reference sidebar which makes them standout compared to the other sections for no real reason.

[ZeroIntensity]

Getting there!

[ZeroIntensity]

Per my complaints with native coroutines in my other comment, I dislike introducing a distinction between "corotuine functions" (async def) and "functions that return coroutines" (e.g., a C function). Please just say "coroutines".

[anordin95]

Hm. I'm not sure I see how this sentence is distinguishing between native coroutines and C-generated coroutines.

Do you want this change:
"coroutine objects" -> "coroutine"

or this one:
"coroutine functions, coroutine objects" -> "coroutines"

[ZeroIntensity]

The second one.

[anordin95]

Ah, I fear I disagree then. I think the distinction between things that generate coroutines, and coroutines is important to make. And I think it's a common gotcha for beginners. I still don't see how this touches on native coroutines vs. C coroutines. Perhaps there's another way to address this?

[ZeroIntensity]

It's an even bigger frustration of mine from working with C coroutines; inspect.iscoroutinefunction is sometime used to check if await something() works (by checking if it's an async def function), which is impossible in the C API.

I'm fine with keeping the distinction later on, but I don't think we should emphasize it in the introduction.

[anordin95]

Are C coroutines something most, typical asyncio users encounter? I want to focus on the concepts that will be most immediately useful.