Stabilize doctest-xcompile

[ehuss]

This stabilizes the doctest-xcompile feature by unconditionally enabling it.

Closes #7040
Closes #12118

What is being stabilized?

This changes it so that cargo will run doctests when using the --target flag for a target that is not the host. Previously, cargo would ignore doctests (and show a note if passing --verbose).

A wrapper for running the doctest can be specified with the target.*.runner configuration option (which is powered by the --test-runtool rustdoc flag). This would typically be something like qemu to run under emulation. It is my understanding that this should work just like running other kinds of tests.

Additionally, the target.*.linker config option is honored for using a custom linker.

Already stabilized in rustdoc is the ability to ignore tests per-target.

Motivation

The lack of doctest cross-compile support has always been simply due to the lack of functionality in rustdoc to support this. Rustdoc gained the ability to cross-compile doctests some time ago, but there were additional flags like the test runner that were not stabilized until just recently.

This is intended to ensure that projects have full test coverage even when doing cross-compilation. It can be surprising to some that this was not happening, particularly since cargo is silent about it.

Risks

The cargo team had several conversations about how to roll out this feature. Ultimately we decided to enable it unconditionally with the understanding that most projects will probably want to have their doctests covered, and that any breakage will be a local concern that can be resolved by either fixing the test or ignoring the target.

Tests in rust-lang/rust run into this issue, particularly on android, and those will need to be fixed before this reaches beta. This is something I am looking into.

Some cross-compiling scenarios may need codegen flags that are not supported. It's not clear how common this will be, or if ignoring will be a solution, or how difficult it would be to update rustdoc and cargo to support these. Additionally, the split between RUSTFLAGS and RUSTDOCFLAGS can be cumbersome.

Implementation history

• Allow cross-compiling doctests rust#60387 -- Support added to rustdoc to support the --target flag and runtool and per-target-ignores.
• Added ability to crosscompile doctests #6892 -- Initial support in cargo.
• Add documentation for the -Zdoctest-xcompile feature #7391 -- Added unstable documentation.
• Fix target for doc test cross compilation #8094 -- Fix target for doc test cross compilation
• Fix doctests not running with --target=HOST. #8358 -- Fixed regression with --target=HOST not working on stable.
• Add a note about doctest xcompile. #10132 -- Added note about doctests not running (under --verbose).
• rustdoc: Fixes with --test-run-directory and relative paths. rust#112751 -- Fixed --test-run-directory interaction with --test-runtool.
• Stabilize flags for doctest cross compilation rust#137096 -- Stabilization (and rename) of the rustdoc --test-runtool and --test-runtool-arg CLI args, and drops --enable-per-target-ignores unconditionally enabling it.

Test coverage

Cargo tests:

• artifact_dep::cross_doctests_works_with_artifacts -- Checks that doctest has access to the artifact dependencies.
• build_script::duplicate_script_with_extra_env -- Checks that build-script env and cfg values are correctly handled on host versus target when cross running doctests.
• cross_compile::cross_tests -- Basic test that cross-compiled tests work.
• cross_compile::doctest_xcompile_linker -- Checks that the linker config argument works.
• custom_target::custom_target_minimal -- Checks that .json targets work with rustdoc cross tests.
• test::cargo_test_doctest_xcompile_ignores -- Checks the ignore-* syntax works.
• test::cargo_test_doctest_xcompile_runner -- Checks runner with cross doctests.
• test::cargo_test_doctest_xcompile_no_runner -- Checks cross doctests without a runner.

Rustdoc tests:

• run-make/doctest-runtool -- Tests behavior of --test-run-directory with relative paths of the runner.
• rustdoc/doctest/doctest-runtool -- Tests for --test-runtool and --test-runtool-arg.

Future concerns

There have been some discussions (rust-lang/testing-devex-team#5) about changing how doctests are driven. My understanding is that stabilizing this should not affect those plans, since if cargo becomes the driver, it will simply need to build things with --target and use the appropriate runner.

Change notes

This PR changed tests a little:

• artifact_dep::no_cross_doctests_works_with_artifacts was changed now that doctests actually work.
• cross_compile::cross_tests was changed to properly check doctests.
• cross_compile::no_cross_doctests dropped since it is no longer relevant.
• standard_lib::doctest didn't need -Zdoctest-xcompile since -Zbuild-std no longer uses a target.
• test::cargo_test_doctest_xcompile was removed since it is a duplicate of cross_compile::cross_tests

I think this should probably wait until the next release cutoff, moving this to 1.89 (will update the PR accordingly if that happens).

[rustbot]

r? @weihanglo

rustbot has assigned @weihanglo.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[ehuss]

@rust-lang/cargo I'm proposing to stabilize this, though I think it should wait until 1.89. I figured I'd get this process started in case anyone had any concerns or needs time to look at it.

@rfcbot fcp merge

[rfcbot]

Team member @ehuss has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @Muscraft
• @Rustin170506
• @arlosi
• @ehuss
• @epage
• @joshtriplett
• @weihanglo

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[weihanglo]

why 1.87 not 1.88?

[ehuss]

Because this was written a long time ago, and I missed updating this line. 😆

[weihanglo]

but looks like we'll miss 1.88 after 10-day FCP 😬

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[weihanglo]

@ehuss feel free to merge it after updating all 1.88 occurrences to 1.89

[RalfJung]

FYI, this has been causing us some trouble in Miri: we need rustdoc to invoke Miri as the runner in a consistent way. When Miri gets tested in the rustc test suite, that is using beta cargo, but for our users out there it will be nightly cargo.

What we used to do is to just set --test-runtool ourselves (which we can do since cargo invokes cargo-miri as rustdoc binary, and then we do some flag adjustments before invoking the actual rustdoc). That broke when cargo started to set that flag, once this PR landed (we also set target.'cfg(all())'.runner so that Miri is used to run regular tests). So instead I made it so we rely on cargo setting that flag. But then I had to add -Zdoctest-xcompile to ensure Miri's tests still pass in ./x... and then the tests failed in Miri's repo because of the warning that the flag is now stable (and also all our users will see that warning until the next subtree bump). What I ended up doing is use some env vars to detect whether we run in ./x or not and then decide whether to pass -Zdoctest-xcompile or not -- I hope that will work.

Not sure what the less to learn here is, but it would have been good to have some way to consistently invoke old and new cargo such that it behaves in exactly the same way, e.g. by having some way to suppress the warning that using the flag is a NOP as the feature is already stable.

[weihanglo]

Sorry about the situation Ralf 😞.

e.g. by having some way to suppress the warning that using the flag is a NOP as the feature is already stable.

Cargo now has its own (unstable) linting system, and feel free to propose new lint via this issue template! It is currently limited to handle only Cargo.toml, but I believe eventually it will expand to config.toml and other areas.

but it would have been good to have some way to consistently invoke old and new cargo such that it behaves in exactly the same way

Might not be applicable to this, though the team discussed how to test the compatibility between old and new cargoes. If you have some other ideas, feel free to share!