Run Sphinx builds in isolated subprocesses

[kehach07]

Fixed issue :- Docs intermittently mix translations of Sphinx labels #2416

Run each Sphinx build in a separate subprocess to avoid global state leakage
between documentation builds.

[pauloxnet]

I would suggest removing old lines of code instead of commenting.

[kehach07]

I would suggest removing old lines of code instead of commenting.

thank you for the feedback, i have implemented the suggestion.

[medmunds]

This seems to me like a safer way to use Sphinx, and removing the monkeypatching and undocumented Sphinx internals feels much cleaner.

One important test would be whether this generates the same built docs as before. (I'm not set up to run djangoproject.com locally, and it doesn't seem like the PR checks deploy a preview site.)

FWIW, Copilot had some concerns:

Summary — will the built docs match previous builds (aside from the bug being fixed)?

Not with the code as written. The subprocess approach is the right direction (it properly isolates Sphinx/docutils global state), but the current way extensions are passed to sphinx-build is likely to change behaviour and break loading of extensions, so output may differ. Other behavioral changes (single-threaded build, doctree location) are less likely to change rendered content, but could affect build speed and doctree reuse.

The point about extensions seems valid: they need to be full dotted paths in a command line arg, and there are potential shell quoting issues. But I question some of its other concerns: e.g., preventing doctree reuse was intentional. (Also, the AI mixing British and American spelling in the same paragraph doesn't inspire confidence 😄 .)

Copilot also pointed out logging behavior would change.

[To clarify: I'm not a maintainer on this project, just a human who uses docs.djangoproject.com and reported a bug.]

[kehach07]

This seems to me like a safer way to use Sphinx, and removing the monkeypatching and undocumented Sphinx internals feels much cleaner.

One important test would be whether this generates the same built docs as before. (I'm not set up to run djangoproject.com locally, and it doesn't seem like the PR checks deploy a preview site.)

FWIW, Copilot had some concerns:

Summary — will the built docs match previous builds (aside from the bug being fixed)?
Not with the code as written. The subprocess approach is the right direction (it properly isolates Sphinx/docutils global state), but the current way extensions are passed to sphinx-build is likely to change behaviour and break loading of extensions, so output may differ. Other behavioral changes (single-threaded build, doctree location) are less likely to change rendered content, but could affect build speed and doctree reuse.

The point about extensions seems valid: they need to be full dotted paths in a command line arg, and there are potential shell quoting issues. But I question some of its other concerns: e.g., preventing doctree reuse was intentional. (Also, the AI mixing British and American spelling in the same paragraph doesn't inspire confidence 😄 .)

Copilot also pointed out logging behavior would change.

[To clarify: I'm not a maintainer on this project, just a human who uses docs.djangoproject.com and reported a bug.]

Thanks for the detailed review!

I’ve updated the PR and the implementation to avoid passing extensions via the command line and now rely entirely on conf.py, so extension loading and rendered output should match previous builds (aside from the bug fix).

The subprocess isolation is intentional to avoid shared Sphinx/docutils state; changes like separate doctrees are expected but shouldn’t affect rendered content. Logging differences are noted, but behaviorally this should now be much closer to the existing builds.

[sarahboyce]

the current way extensions are passed to sphinx-build is likely to change behaviour and break loading of extensions

☝️ these changes currently mean the custom extension is no longer used as it roughly reverts the use added in #1947

How to test this works locally:

• Build the docs (python -m manage update_docs)
• Search for "select"
• You should see code reference search results such as QuerySet.select_for_update (currently with these changes, these results are lost)