don't apply temporary lifetime extension rules to non-extended super let

[dianne]

Reference PR: rust-lang/reference#1980

This changes the semantics for super let (and macros implemented in terms of it, such as pin!, format_args!, write!, and println!) as suggested by @theemathas in #145784 (comment), making super let initializers only count as extending expressions when the super let itself is within an extending block. Since super let initializers aren't temporary drop scopes, their temporaries outside of inner temporary scopes are effectively always extended, even when not in extending positions; this only affects two cases as far as I can tell:

• Block tail expressions in Rust 2024. This PR makes f(pin!({ &temp() })) drop temp() at the end of the block in Rust 2024, whereas previously it would live until after the call to f because syntactically the temp() was in an extending position as a result of super let in pin!'s expansion.
• super let nested within a non-extended super let is no longer extended. i.e. a normal let is required to treat super lets as extending (in which case nested super lets will also be extending).

Closes #145784

This is a breaking change. Both static and dynamic semantics are affected. The most likely breakage is for programs to stop compiling, but it's technically possible for drop order to silently change as well (as in #145784). Since this affects stable macros, it probably would need a crater run.

Nominating for discussion alongside #145784: @rustbot label +I-lang-nominated +I-libs-api-nominated

Tracking issue for super let: #139076

[rustbot]

r? @jackh726

rustbot has assigned @jackh726.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[dianne]

Copied from #145784 (comment), since I think this is a notable caveat of this PR and worth considering before approving it:

This comes with a bit of a gotcha in terms of temporary lifetimes: it might be strange that the temp() would live longer in

non_extending({ let x = { &temp() }; f(x) }); // ok

than in

non_extending({ super let x = { &temp() }; f(x) }); // error

Though the case for if let is similar: its scrutinee isn't a temporary scope and it doesn't have lifetime extension rules that can make block tail expressions' temporaries live longer.

[dianne]

Also copied since it motivates this PR: I think something like this may be necessary for the identity

&EXPR === { super let x = &EXPR; x }

to hold in both extending and non-extending contexts without changing how block tail expression scopes work. Substituting, e.g. { &temp() } in for EXPR, the identity only currently holds in extending contexts in Rust 2024. In non-extending contexts,

&{ &temp() }

drops temp() when leaving the block, but

{ super let x = &{ &temp() }; x }

would extend it to outlive x. This PR shortens the lifetime of temp() such that it's dropped at the end of the block in both cases. I haven't done any rigorous proof or extensive testing that this PR together with #145342 makes the identity always hold, however.

[traviscross]

To confirm, with this PR, does this behavior hold (in Rust 2024)?:

fn f<T>(_: LogDrop<'_>, x: T) -> T { x }

// These two should be the same.
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(2), &{ &raw const *&e.log(1) });
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(2), {
        super let v = &{ &raw const *&e.log(1) };
        v
    });
    drop(e.log(3));
});
// These two should be the same.
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(1), &&raw const *&e.log(2));
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(1), {
        super let v = &&raw const *&e.log(2);
        v
    });
    drop(e.log(3));
});
// These two should be the same.
assert_drop_order(1..=2, |e| {
    let _v = &{ &raw const *&e.log(2) };
    drop(e.log(1));
});
assert_drop_order(1..=2, |e| {
    let _v = {
        super let v = &{ &raw const *&e.log(2) };
        v
    };
    drop(e.log(1));
});

Playground link

(If any of these are missing, please add them as tests.)

[theemathas]

super let nested within a non-extended super let is no longer extended

Does this PR affect any edition 2021 code?

[dianne]

To confirm, with this PR, does this behavior hold (in Rust 2024)?:

Those all hold under this PR, yes. I've added them all as tests (with some additional versioning to account for the Edition-dependent drop order in the first one); thanks!

super let nested within a non-extended super let is no longer extended

Does this PR affect any edition 2021 code?

Not that I'm aware of. That detail matters in Edition 2024, since the nested super let could have an extending borrow operator within a block tail expression in its initializer, which would previously extend the borrowed temporary's lifetime to outlive the outer super let's binding. In Edition 2021, that temporary outlives the outer super let's binding regardless, since block tail expressions (and super let initializers) aren't temporary scopes.

[traviscross]

Suggested change

	// We have extending borrow expressions within the initializer
	// expression.
	// We have extending borrow expressions within a non-extending
	// expression within the initializer expression.

(Revising my earlier text here.)

[traviscross]

This is correct, I believe. Let's propose to do it.

@rfcbot fcp merge

@dianne, let's document this in the Reference.

cc @rust-lang/lang-docs

[rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[traviscross]

cc @m-ou-se @rust-lang/libs-api

[dianne]

The real-looking failures seem to involve something like format!("{something}{}", if cond { "str" } else { &format!(...) }). This is the regression we were expecting (#145838 (comment), #145838 (comment)), though it's interesting that it's manifested in such a specific and consistent way: the otherwise-unnecessary & operator is allowing deref coercion to happen.

Is a crater retry run needed, due to the number of spurious results?

There's a lot (186071) of prepare-fail crates, yeah. I think at least one of the real regressions is in there, since it showed up as a faulty dep but not in the normal regressed list.

---

Edit: Looking at the test failures, it seems like there's a lot of tests for nondeterministic behavior (lots in particular relying on timing or random numbers). I don't have time to go through them all at the moment, but I can try opening a PR against the denylist later (or a few if breaking them up would help in reviewing). I imagine these are the sorts of things that should be skip-tests = true on there?

[dianne]

This may be on the severe side, but instead of the approach this PR takes, I wonder if it would make sense to apply some form of lifetime extension to extending borrow expressions within block tail expressions, regardless of whether they appear in let statement initializers. It feels unintuitive to me that it's context-dependent¹ whether borrows that look like they should be extending outlive block tail expressions. You can't even just check if the immediately surrounding context is a let, both since it could be in a deeply-nested extending expression (possibly involving macro expansions) and because not every appearance of the let keyword uses the same lifetimes for the scrutinee's temporaries (looking at if let here; I imagine the difference there isn't immediately obvious to users).

Is there any case where { &EXPR } (or { Struct { field: &EXPR } }, { if cond { &EXPR1 } else { &EXPR2 } }, etc.) would cause something to live too long in a non-extending context, but it's fine if it's extended in an extending context?

Slightly more precisely, I'm proposing that the grammar for extending expressions could serve two purposes:

• Extending let statement initializers' borrowed temporaries to outlive the bindings.
• In Rust 2024, extending borrowed temporaries in block tail expressions to outlive the block's result.

I feel like this fits naturally into Rust's existing model of temporary lifetimes. In both cases, it's about extending the result of the temporary to match its use when the result value is known syntactically to only be okay to use² if the reference is live. It also parallels the two cases for super let's bindings' lifetimes: in a let statement initializer, super let's bindings outlive the let's bindings, and otherwise they outlive the result of their block.

Sometime later I'll try throwing together a PR to experiment with this.

Logistics-wise, I think that approach wouldn't need to be backported. I'll have to experiment of course (and make sure those crates that regressed under this PR don't regress under it), but my hope is it wouldn't break real code, so it could be left to cook as long as needed without accumulating regressed crates.

Footnotes

1. To be clear, the part I find unintuitive is that the lifetime of the temp() in { &temp() } changes depending on where it's used, because the borrow is extending when the block is an extending expression, and otherwise it's dropped in the enclosing temporary scope (the block tail expression). Behavior is at least consistent if you look "up" from a borrow and find a non-extending expression: the temp() in function(&temp()) is always dropped at the end of the enclosing temporary scope. ↩

2. Using informal terminology here to account for dangling pointers: { &temp() as *const T } compiles but is UB to deref if it appeared in a non-extending position. As such, extending the lifetime of temp() technically changes the runtime semantics of some currently-compiling programs. You'd need to write something like that, depend on exactly when temp() is dropped, and then ignore the result of the block. ↩

[traviscross]

Interesting point. Our current order here is rather challenging to explain because one can't work from the inside out.

Relatedly, can you explain the behavior of "3"?:

assert_drop_order(1..=3, |e| { //~ 1.
    &({ &raw const *&e.log(1) }, drop(e.log(2)));
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| { //~ 2.
    { let _x; _x = &({ &raw const *&e.log(1) }, drop(e.log(2))); }
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| { //~ 3.
    _ = &({ &raw const *&e.log(2) }, drop(e.log(1)));
    drop(e.log(3));
});

Playground link

[dianne]

I believe that's due to destructuring assignment expression desugaring. Since a pattern is being assigned to in an assignment expression, it actually turns into a let statement in the HIR to perform the destructuring, then an assignment expression is generated for each binding present. In this case, there's no bindings, so no assignment expressions are present in the final HIR.

assert_drop_order(#[lang = "range_inclusive_new"](1, 3), //~ 3.
    |e|
        {
            { let _ = &({ &raw const *&e.log(2) }, drop(e.log(1))); };
            drop(e.log(3));
        });

It's wrapped in a block, as above, which prevents temporaries from living past the end of the desugared destructuring assignment, but it does extend e.log(2) to live past the initializer's evaluation. Thus drop(e.log(1)) happens first.

[traviscross]

Interesting, makes sense. E.g., it does the same thing with:

assert_drop_order(1..=3, |e| { //~ 5.
    let _x; let _y;
    (_x, _y) = ({ &raw const *&e.log(2) }, drop(e.log(1)));
    drop(e.log(3));
});

We do document the desugaring of destructing assignments in the Reference, but we don't note anywhere the effect this has on drop order. Probably we should do that.

cc @rust-lang/lang-docs

This is, I think, an argument in favor of the semantic that you're proposing, as that would make the drop order between these cases more similar (as I understand your proposal). What do you think, would it make the drop order of temporaries in _ = $EXPR; and drop($EXPR); exactly the same, or would there be remaining differences?

[dianne]

I think they'd be the exactly the same. The same rules would be used to determine the temporaries that live to the end of the block in { let _ = $EXPR; } and those that live to the end of the statement in drop($EXPR);.

[jieyouxu]

Retrying the prepared-fails.
@craterbot run mode=build-and-test crates=https://crater-reports.s3.amazonaws.com/pr-145838/retry-regressed-list.txt p=1

[craterbot]

👌 Experiment pr-145838-1 created and queued.
🤖 Automatically detected try build b83b707
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[traviscross]

We'd be breaking, e.g.:

fn f(x: *const u8, y: *mut u8) -> u8 {
    unsafe { y.write(42); x.read() }
}

fn main() {
    let x = core::cell::RefCell::new((0u8, 0u8));
    // Sound under TB.
    f({ &raw const x.borrow().0 }, &raw mut x.borrow_mut().1);
}

...not that anyone would defend that code, I hope. I wonder if it'd break anything more defensible.

[jieyouxu]

Edit: Looking at the test failures, it seems like there's a lot of tests for nondeterministic behavior (lots in particular relying on timing or random numbers). I don't have time to go through them all at the moment, but I can try opening a PR against the denylist later (or a few if breaking them up would help in reviewing). I imagine these are the sorts of things that should be skip-tests = true on there?

Yes. Please do feel free to submit skip-tests for crates with non-det tests, you can cc me on the crater PR. Smaller PRs would make it easier, but not too fussed in any case.

[traviscross]

I'd suggest to go ahead and still submit PRs to the affected projects. Even if we might want to consider this new semantic, we may in the meanwhile not want to be accumulating code in the ecosystem that relies on this accidental divergence between the intended and the current behavior of super let. Even if we like the new semantic, it could always happen that we find some reason many releases from now that prevents us from stabilizing it, and then it'd be harder to close the divergence in the way this PR does.

[dianne]

We'd be breaking, e.g.:

Oh, true. I'll have to think on that. I'm not that familiar with what sorts of things are realistic in unsafe code though, so I may not be cut out to evaluate it entirely on my own.

---

Speaking of breakages due to lifetime changes, I don't think it's been mentioned yet, but I think this PR can silently introduce UB in cases where raw pointers' pointee lifespans are shortened. e.g. if anyone wrote

// Implicitly coerce to sneakily create a dangling pointer with `&`:
non_extending(pin!(if cond { raw_ptr } else { &temp() }));

or

non_extending(pin!({ Struct { raw_ptr: &temp() } }));

temp() wouldn't live past the block under this PR, so the pointer would be dangling. Currently (and under the alternative semantics I proposed), temp() is extended to outlive the result of pin!. One might expect it to be, too, since it is extended to live long enough under this PR if the pin! is extending:

let pin1 = pin!(if cond { raw_ptr } else { &temp() });
let pin2 = pin!({ Struct { raw_ptr: &temp() } });

We've talked about the case where this shortening causes compilation failures (due to borrow-checking), but it totally slipped my mind that people could be relying on subtle lifetime extension behavior in unsafe code like that.

[theemathas]

I've reviewed 13 of the test failures in #145838 (comment) (top-to-bottom from Arxoto.sudoku.78ccf4297d22f37e62e8bf07ff19f7f2e7843139 to LegendarySaiyan.structures.f375d1f242c501951f7a3a6c49a09a7895f57326). All of them are flaky tests. Made a crater PR.

[theemathas]

Saethlin told me that I should wait for the retry before checking the test failures.

[craterbot]

🚧 Experiment pr-145838-1 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-145838-1 is completed!
📊 697 regressed and 670 fixed (187804 total)
📊 580 spurious results on the retry-regessed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Footnotes

1. re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-145838-1/retry-regressed-list.txt ↩

[dianne]

At a glance, the "build compiler error" failures look real, following the pattern of using &format!(...) in a block of a conditional expression inside a format argument (alongside at least one other format argument). The rest I've peeked at look spurious. The "fixed" crates are almost certainly all noise, so I'm expecting a comparable number of spurious regressions. I'll try and review them more comprehensively later.

---

In the mean time, I've implemented my lifetime extension suggestion for block tails. It's not ready for review yet, but it lives at #146098 now for testing/discussion purposes.

[jieyouxu]

Retrying since there's a lot of "cannot find directory"s.
@craterbot run mode=build-and-test crates=https://crater-reports.s3.amazonaws.com/pr-145838-1/retry-regressed-list.txt p=1

[craterbot]

👌 Experiment pr-145838-2 created and queued.
🤖 Automatically detected try build b83b707
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more