[rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition #143900

GuillaumeGomez · 2025-07-13T18:52:32Z

Fixes #143009.
Fixes #143858.

Since it includes fixes from #143453, it's taking it over (commits 2, 3 and 4 are from #143453).

For --no-run, we forgot to check the "global" options in the 2024 edition, fixed in the first commit.

For should_panic fix, the exit code check has been fixed.

cc @TroyKomodo (thanks so much for providing such a complete test, made my life a lot easier!)
r? @notriddle

rustbot · 2025-07-13T18:52:40Z

This PR modifies run-make tests.

cc @jieyouxu

purplesyringa · 2025-07-14T02:57:10Z

For should_panic fix, instead of checking the exit code, we directly wrap the doctest code with catch_unwind and if a panic happened, then we return success for the test, otherwise we display the appropriate error message.

I... do not see anything like that in the patch? There's two checks for exit code 101.

GuillaumeGomez · 2025-07-14T10:17:00Z

Woups, copied/pasted comment from original PR which was outdated after discussion with you (on the previous PR). ^^'

EDIT: Updated comment.

purplesyringa · 2025-07-14T10:26:41Z

if langstr.should_panic {
    if out.status.code() == Some(101) {
        return Ok(());
    } else if out.status.success() {
        return Err(TestFailure::UnexpectedRunPass);
    }
}
if !out.status.success() {
    return Err(TestFailure::ExecutionFailure(out));
}

Do you think something like this would make sense, so that it's easier for the user to differentiate between aborts and panic-less success?

GuillaumeGomez · 2025-07-14T10:29:30Z

TestFailure::UnexpectedRunPass displays "Test didn't panic, but it's marked `should_panic`.", which means we'd need to add a new case. Do you think it's necessary to differentiate between success and non-panic failures?

purplesyringa · 2025-07-14T10:31:43Z

Wouldn't changing UnexpectedRunPass to spell "Test executable succeeded, but it's marked should_panic." (i.e. rolling back the change in this PR) work? After all, if the change was to let UnexpectedRunPass represent non-panicking failures, it won't be necessary if we just use ExecutionFailure for that.

GuillaumeGomez · 2025-07-14T10:32:54Z

That wouldn't cover exit(1) anymore if we did so.

purplesyringa · 2025-07-14T10:34:19Z

I've amended my comment a moment before you replied, apologies.

GuillaumeGomez · 2025-07-14T10:37:44Z

Replied too fast then. 😅

The message for ExecutionFailure is too general imo ("Test executable failed"). It doesn't help to understand should_panic didn't panic. I think once this PR is merged, you can send a new one to create a new variant in case it failed but didn't panic, like that we can have a message that is covering this case. What do you think?

purplesyringa · 2025-07-14T10:39:38Z

The message for ExecutionFailure is too general imo ("Test executable failed"). It doesn't help to understand should_panic didn't panic.

I think we're talking past each other. I suggest that:

If the program didn't panic and exited with code 0, we keep using UnexpectedRunPass ("test didn't panic"). This is the main use case for should_panic and has a perfectly legible description.
If the program didn't panic and exited with a non-zero code, we use ExecutionFailure ("executable failed"). I believe that it's highly unlikely this failure mode of a should_panic test can be attributed to anything but UB or an abort, exactly like with non-should_panic tests, so I don't think it's necessary to specify that it's happened in a should_panic test specifically.

Wouldn't that work?

GuillaumeGomez · 2025-07-14T10:43:18Z

I still think it's too general. For end users, should_panic means the execution is supposed to fail. However it might be confusing to get "executable failed" when it's exactly what's expected (well, except it was expecting a panic and it's different). Hence why I think the current message is better and why, if we go down this road, we should add a new case to handle this case instead of relying on the too generated ExecutionFailure.

purplesyringa · 2025-07-14T10:44:24Z

That makes sense, yeah. Let's delay it until after this PR lands then.

bors · 2025-07-30T22:29:39Z

☔ The latest upstream changes (presumably #144692) made this pull request unmergeable. Please resolve the merge conflicts.

…Amanieu Make `libtest::ERROR_EXIT_CODE` const public to not redefine it in rustdoc I think it's better to make this constant public so it can be used by crates using `libtest` as dependency. As a side-note, I will update rust-lang#143900 to make use of this constant once this is current PR is merged.

Rollup merge of #144297 - GuillaumeGomez:ERROR_EXIT_CODE, r=Amanieu Make `libtest::ERROR_EXIT_CODE` const public to not redefine it in rustdoc I think it's better to make this constant public so it can be used by crates using `libtest` as dependency. As a side-note, I will update #143900 to make use of this constant once this is current PR is merged.

lolbinarycat

Everything looks correct to me, just some very minor nitpicks

lolbinarycat · 2025-07-31T17:44:31Z

tests/run-make/rustdoc-should-panic/rmake.rs

+    for text in should_contain {
+        assert!(
+            output.contains(text),
+            "output doesn't contains (edition: {edition}) {:?}\nfull output: {output}",


Suggested change

"output doesn't contains (edition: {edition}) {:?}\nfull output: {output}",

"output (edition: {edition}) doesn't contains {:?}\nfull output: {output}",

tiny nitpick: i think this order makes a bit more sense, otherwise it feels like it's saying "doesn't contain edition"

also: "doesn't contain", not "doesn't contains".

lolbinarycat · 2025-07-31T17:47:52Z

src/librustdoc/doctest.rs

@@ -827,7 +827,7 @@ fn run_test(
    match result {
        Err(e) => return (duration, Err(TestFailure::ExecutionError(e))),
        Ok(out) => {
-            if langstr.should_panic && out.status.success() {
+            if langstr.should_panic && out.status.code() != Some(101) {


Suggested change

if langstr.should_panic && out.status.code() != Some(101) {

// FIXME: use test::ERROR_EXIT_CODE once public

if langstr.should_panic && out.status.code() != Some(101) {

lolbinarycat · 2025-07-31T17:48:15Z

src/librustdoc/doctest/runner.rs

        let out = std::process::Command::new(bin)
            .env(self::RUN_OPTION, test_nb.to_string())
            .args(std::env::args().skip(1).collect::<Vec<_>>())
            .output()
            .expect(\"failed to run command\");
-        if !out.status.success() {{
+        if should_panic {{
+            if out.status.code() != Some(101) {{


Suggested change

if out.status.code() != Some(101) {{

// FIXME: use test::ERROR_EXIT_CODE once public

if out.status.code() != Some(101) {{

GuillaumeGomez · 2025-08-01T13:22:43Z

Thanks to both of you!

@bors r=lolbinarycat,fmease rollup

bors · 2025-08-01T13:22:46Z

📌 Commit 61c5343 has been approved by lolbinarycat,fmease

It is now in the queue for this repository.

…narycat,fmease [rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition Fixes rust-lang#143009. Fixes rust-lang#143858. Since it includes fixes from rust-lang#143453, it's taking it over (commits 2, 3 and 4 are from rust-lang#143453). For `--no-run`, we forgot to check the "global" options in the 2024 edition, fixed in the first commit. For `should_panic` fix, the exit code check has been fixed. cc `@TroyKomodo` (thanks so much for providing such a complete test, made my life a lot easier!) r? `@notriddle`

Rollup of 10 pull requests Successful merges: - #132748 (get rid of some false negatives in rustdoc::broken_intra_doc_links) - #135771 ([rustdoc] Add support for associated items in "jump to def" feature) - #143360 (loop match: error on `#[const_continue]` outside `#[loop_match]`) - #143662 ([rustdoc] Display unsafe attrs with edition 2024 `unsafe()` wrappers.) - #143900 ([rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition) - #144614 (Fortify RemoveUnneededDrops test.) - #144703 ([test][AIX] ignore extern_weak linkage test) - #144738 (Remove the omit_gdb_pretty_printer_section attribute) - #144756 (detect infinite recursion with tail calls in ctfe) - #144766 (Add human readable name "Cygwin") r? `@ghost` `@rustbot` modify labels: rollup

…narycat,fmease [rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition Fixes rust-lang#143009. Fixes rust-lang#143858. Since it includes fixes from rust-lang#143453, it's taking it over (commits 2, 3 and 4 are from rust-lang#143453). For `--no-run`, we forgot to check the "global" options in the 2024 edition, fixed in the first commit. For `should_panic` fix, the exit code check has been fixed. cc ``@TroyKomodo`` (thanks so much for providing such a complete test, made my life a lot easier!) r? ``@notriddle``

Rollup of 11 pull requests Successful merges: - #132748 (get rid of some false negatives in rustdoc::broken_intra_doc_links) - #135771 ([rustdoc] Add support for associated items in "jump to def" feature) - #143360 (loop match: error on `#[const_continue]` outside `#[loop_match]`) - #143662 ([rustdoc] Display unsafe attrs with edition 2024 `unsafe()` wrappers.) - #143900 ([rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition) - #144478 (Improve formatting of doc code blocks) - #144703 ([test][AIX] ignore extern_weak linkage test) - #144747 (compiletest: Improve diagnostics for line annotation mismatches 2) - #144756 (detect infinite recursion with tail calls in ctfe) - #144766 (Add human readable name "Cygwin") - #144782 (Properly pass path to staged `rustc` to `compiletest` self-tests) r? `@ghost` `@rustbot` modify labels: rollup

GuillaumeGomez · 2025-08-01T21:15:24Z

Failed in #144791.

@bors r-

… typo

rustbot · 2025-08-01T21:43:58Z

⚠️ Warning ⚠️

There are issue links (such as #123) in the commit messages of the following commits.
Please move them to the PR description, to avoid spamming the issues with references to the commit, and so this bot can automatically canonicalize them to avoid issues with subtree.
- cc60a24
- 0a01e8e

GuillaumeGomez · 2025-08-01T21:44:23Z

Just tested locally and it passed fine. Might be another PR in the rollup I guess.

@bors r=lolbinarycat,fmease rollup

bors · 2025-08-01T21:44:25Z

📌 Commit 7b8015c has been approved by lolbinarycat,fmease

It is now in the queue for this repository.

rustbot assigned notriddle Jul 13, 2025

This comment has been minimized.

Sign in to view

This was referenced Jul 13, 2025

Correctly handle should_panic doctest attribute #143453

Closed

Rustdoc --no-run runs when --edition=2024 is provided #143858

Open

fmease self-assigned this Jul 17, 2025

GuillaumeGomez mentioned this pull request Jul 22, 2025

Make libtest::ERROR_EXIT_CODE const public to not redefine it in rustdoc #144297

Merged

GuillaumeGomez force-pushed the fix-no-run branch from 72aadc6 to 248c61c Compare July 31, 2025 09:00

This comment has been minimized.

Sign in to view

GuillaumeGomez force-pushed the fix-no-run branch from 248c61c to c365e6e Compare July 31, 2025 10:07

This comment has been minimized.

Sign in to view

lolbinarycat reviewed Jul 31, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 1, 2025

GuillaumeGomez mentioned this pull request Aug 1, 2025

Rollup of 10 pull requests #144784

Closed

GuillaumeGomez mentioned this pull request Aug 1, 2025

Rollup of 11 pull requests #144791

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 1, 2025

GuillaumeGomez added 6 commits August 1, 2025 23:17

Correctly handle --no-run rustdoc test option

f618a05

Correctly handle should_panic doctest attribute

202a05b

Add regression test for rust-lang#143009

cc60a24

Update std doctests

d67a4a5

Add regression test for rust-lang#143858

0a01e8e

Add FIXME comments to use test::ERROR_EXIT_CODE once public and fix…

7b8015c

… typo

GuillaumeGomez force-pushed the fix-no-run branch from 61c5343 to 7b8015c Compare August 1, 2025 21:43

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 1, 2025

	"output doesn't contains (edition: {edition}) {:?}\nfull output: {output}",
	"output (edition: {edition}) doesn't contains {:?}\nfull output: {output}",

	if langstr.should_panic && out.status.code() != Some(101) {
	// FIXME: use test::ERROR_EXIT_CODE once public
	if langstr.should_panic && out.status.code() != Some(101) {

	if out.status.code() != Some(101) {{
	// FIXME: use test::ERROR_EXIT_CODE once public
	if out.status.code() != Some(101) {{

[rustdoc] Correctly handle should_panic doctest attribute and fix --no-run test flag on the 2024 edition #143900

Are you sure you want to change the base?

[rustdoc] Correctly handle should_panic doctest attribute and fix --no-run test flag on the 2024 edition #143900

Conversation

GuillaumeGomez commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 13, 2025

Uh oh!

This comment has been minimized.

purplesyringa commented Jul 14, 2025

Uh oh!

GuillaumeGomez commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

purplesyringa commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeGomez commented Jul 14, 2025

Uh oh!

purplesyringa commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeGomez commented Jul 14, 2025

Uh oh!

purplesyringa commented Jul 14, 2025

Uh oh!

GuillaumeGomez commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

purplesyringa commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeGomez commented Jul 14, 2025

Uh oh!

purplesyringa commented Jul 14, 2025

Uh oh!

bors commented Jul 30, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lolbinarycat left a comment

Choose a reason for hiding this comment

Uh oh!

lolbinarycat Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

fmease Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

lolbinarycat Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

lolbinarycat Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomez commented Aug 1, 2025

Uh oh!

bors commented Aug 1, 2025

Uh oh!

GuillaumeGomez commented Aug 1, 2025

Uh oh!

rustbot commented Aug 1, 2025

Uh oh!

GuillaumeGomez commented Aug 1, 2025

Uh oh!

bors commented Aug 1, 2025

Uh oh!

Uh oh!

[rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition #143900

[rustdoc] Correctly handle `should_panic` doctest attribute and fix `--no-run` test flag on the 2024 edition #143900

GuillaumeGomez commented Jul 13, 2025 •

edited

Loading

GuillaumeGomez commented Jul 14, 2025 •

edited

Loading

purplesyringa commented Jul 14, 2025 •

edited

Loading

purplesyringa commented Jul 14, 2025 •

edited

Loading

GuillaumeGomez commented Jul 14, 2025 •

edited

Loading

purplesyringa commented Jul 14, 2025 •

edited

Loading