Skip to content

opt: preserve mutation rows in RETURNING subquery joins - #174790

Open
Alignyx wants to merge 2 commits into
cockroachdb:masterfrom
Alignyx:fix-105640-mutation-join-multiplicity
Open

opt: preserve mutation rows in RETURNING subquery joins#174790
Alignyx wants to merge 2 commits into
cockroachdb:masterfrom
Alignyx:fix-105640-mutation-join-multiplicity

Conversation

@Alignyx

@Alignyx Alignyx commented Sep 6, 2026

Copy link
Copy Markdown

Fixes #105640.

INSERT ... ON CONFLICT ... RETURNING can successfully write a row while returning no rows when a correlated subquery reads the target table. For example, on an empty table:

CREATE TABLE kv (k INT PRIMARY KEY, v INT);

INSERT INTO kv VALUES (1, 10)
ON CONFLICT (k) DO UPDATE SET v = 10
RETURNING k, v, (SELECT (k, v) FROM kv foo WHERE foo.k = kv.k);

The expected result is one row, (1, 10, NULL): the inserted values are available through RETURNING, while the subquery's table scan does not see the newly inserted row. Before this change, the statement returns zero rows even though a subsequent SELECT finds (1, 10) in the table. The issue reports this behavior in v21.1 and later.

The incorrect inference is in filtersMatchAllLeftRows. Column metadata on a mutation's output can identify the same base-table columns as a scan, but that does not establish that the scan contains a matching row. Treating that metadata as a row-preservation guarantee allows an outer join needed for the subquery to become an inner join, dropping the mutation's result when the scan is empty.

The change is split into two commits:

  1. Restrict the self-join preservation shortcut to inputs whose relational properties have CanMutate == false. Add regressions for insertion into an empty table and the conflict-update path, checking both returned values and stored data. The conflict case also checks that the subquery observes the previous value (1,77) while RETURNING exposes the updated value 10.
  2. Apply the same restriction to the foreign-key preservation shortcuts, both with equality filters and without filters. Reviewing the fallthrough after the self-join check exposed a second reproduction: change the declaration to k INT PRIMARY KEY REFERENCES kv(k) and run the same INSERT. The self-join restriction alone still returns zero rows because the foreign-key shortcut reaches the same invalid conclusion. A foreign-key constraint on stored rows does not establish that a mutation's output has a match in the subquery's scan. Add the self-referencing regression, an uncorrelated RETURNING subquery on a self-referencing table, and a read-only self-join control containing NULL values.

Both join inputs are checked because either may contain a mutation. CanMutate propagates through relational expressions, so the restriction also applies when the mutation is below another operator. Returning false here declines an unsupported proof and keeps the required outer-join behavior. The independent cross-join proof based on a guaranteed nonempty right input remains valid and is retained. Read-only inputs continue to use the existing self-join and foreign-key proofs. This is conservative for expressions containing an unrelated mutation: it may forgo an optimization opportunity; no performance benchmark was run.

Validation:

  • The original regression fails on the unpatched base and passes with the first commit. The self-referencing foreign-key regression still fails with that first commit and passes with the second. The complete returning SQL logic fixture passes on the final change in the single-node local configuration. The additions contain nine result assertions, including checks of stored data.
  • 84 runtime checks pass: the original query and 11 equivalent forms across six initial table states, plus 12 additional cases covering other mutation forms, self-referencing foreign keys, read-only behavior, and scalar-subquery cardinality errors. The expected observations were checked against PostgreSQL 16.13. All 83 successful statements also pass stored-state assertions; the error case verifies SQLSTATE 21000.
  • Both commits independently build and pass the SQL logic regression and the complete opt/memo, opt/norm, and opt/optbuilder package test executables: 220, 46, and 108 passing test/subtest entries respectively, with no failures or skips.
  • crlfmt -fast -tab 2 and git diff --check pass for both commits.

The package executables were built with Bazel's --config=test and run directly with their runfiles. The SQL fixture was run through logictest.RunLogicTest. The ./dev generate, ./dev lint --short, and ./dev test entry points reject the root user in this environment, so those checks and the full repository test suite are not claimed as passing.

A correlated subquery in INSERT ... ON CONFLICT ... RETURNING could lose
all result rows. The self-join matching shortcut treated mutation output
columns as proof that matching rows already existed in a scan of the same
table. That proof is invalid for newly inserted values visible through
RETURNING, and could turn the required outer join into an inner join.

Apply the self-join shortcut only when neither input can mutate. Add SQL
logic coverage for the original RETURNING behavior and stored table data.
The added regression fails on the unpatched source and passes with this
commit.

An independent post-hoc review identified a related foreign-key proof
that needs an additional correction, provided in the next commit.

See also: cockroachdb#105640
Epic: none

Release note (bug fix): Fixed a bug where INSERT ... ON CONFLICT ...
RETURNING with a correlated subquery reading the target table could omit
returned rows even though the write succeeded.
Guarding the self-join shortcut alone leaves another invalid proof in
filtersMatchAllLeftRows. With a self-referencing primary-key foreign key,
the same INSERT ... ON CONFLICT ... RETURNING query falls through to the
foreign-key shortcut and still loses its result row. A constraint on
stored rows does not prove that mutation output has a matching row in the
snapshot read by the RETURNING subquery.

Reject foreign-key preservation proofs when either input can mutate, for
both equality joins and cross joins. Retain the independent cardinality
proof for cross joins whose right input is guaranteed to be nonempty, and
leave the existing proofs available for read-only inputs.

Add a regression for the self-referencing foreign-key reproduction, with
an assertion on the stored row. Also cover an uncorrelated RETURNING
subquery on a self-referencing table and a read-only self-join with NULL
values. The foreign-key reproduction fails with the preceding commit and
passes with this correction.

Fixes cockroachdb#105640
Epic: none

Release note (bug fix): Fixed missing rows in INSERT ... ON CONFLICT ...
RETURNING results when a subquery reads a self-referencing foreign-key
table. The write could succeed while an incorrect join optimization
removed the returned row.
@Alignyx
Alignyx requested a review from a team as a code owner September 6, 2026 15:24
@Alignyx
Alignyx requested review from DrewKimball and removed request for a team September 6, 2026 15:24
@blathers-crl

blathers-crl Bot commented Sep 6, 2026

Copy link
Copy Markdown

Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR.

Before a member of our team reviews your PR, I have some potential action items for you:

  • We notice you have more than one commit in your PR. We try break logical changes into separate commits, but commits such as "fix typo" or "address review commits" should be squashed into one commit and pushed with --force
  • When CI has completed, please ensure no errors have appeared.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl Bot added the O-community Originated from the community label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

O-community Originated from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sql: no rows returned in INSERT .. ON CONFLICT with subquery in RETURNING clause

1 participant