Nightly cassert failures 2026-08-12..17 - consolidated triage (5 issues, 5 root causes, 0 assertions)

[github-actions]

Nightly full-cassert run failed.

Run: https://github.com/citusdata/citus/actions/runs/31990816303

Failing jobs:

• cassert PG18 - regress

First runs are EXPECTED red as pre-existing PG-core and Citus asserts surface.

[ihalatci]

Nightly cassert triage — consolidated analysis (2026-08-12 → 2026-08-17)

This issue now tracks all nightly cassert failures from 2026-08-12 through 2026-08-17.
#8749, #8768, #8769 and #8775 have been closed as duplicates and folded in here.

TL;DR

• 5 issues, 7 failing jobs, but only 5 distinct root causes — and only 3 are actual Citus problems.
• All 5 runs were on the identical commit 52663ed4346d293f9561c1ca7bf3533cd6a11d9d on main.
  No code changed between nights, so every night-to-night difference is non-determinism, not regression.
• Not a single assertion fired. Grepping every available job log for TRAP:, Failed Assertion,
  SIGABRT, signal 6 and server process ... was terminated returned zero hits. Despite this being
  the cassert nightly, no Assert() was tripped on any night — every red job is either a plain output
  diff or GitHub-runner infrastructure.

The 7 failures → 5 causes

	Issue(s)	Failing job	Root cause	Category
A	#8749, #8775, #8776	PG17 / PG18 regress	single_node_enterprise — one extra connection opened	Citus flake
B	#8768	PG16 regress	background_rebalance_parallel_reference_tables — non-deterministic expected output	Citus flake
C	#8775	PG16 regress	statement_cancel_error_message — wrong cancel reason	Real bug
D	#8749	PG17 arbitrary-configs-1	mitmproxy git clone TLS failure	CI infra
E	#8769	PG17 arbitrary-configs-3	hosted runner lost communication	CI infra

---

A — single_node_enterprise: extra connection (3 of the 5 issues)

Byte-identical diff on all three nights, on both PG17 and PG18 — so it is PG-version independent and
reproducible in character rather than random noise.

In the block headed "followed by a multi-shard command"
(src/test/regress/expected/single_node_enterprise.out, ~lines 461-482), SELECT count(*) FROM test;
fans out to 6 shards. Expected output has all 6 issued on already-open connections. Actual output inserts,
immediately before the 6th:

+NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 56, '<ts>');

plus a matching extra +NOTICE: issuing ROLLBACK at the end of the block. The adaptive executor opened one
additional connection for the last shard instead of reusing an existing one — a timing race in connection
management, most likely dependent on citus.executor_slow_start_interval / pool-size behaviour.

This single flake is responsible for three of the five issues.

B — background_rebalance_parallel_reference_tables: non-deterministic baseline

Two symptom classes in one diff:

1. DEBUG: skipping child tables for relation named: table1_colg1 (expected) vs table2_colg1 (actual).
   That DEBUG comes from ColocatedNonPartitionShardIntervalList() at
   src/backend/distributed/utils/colocation_utils.c:1211, printing get_rel_name(distributedTableId).
   Which relation represents a colocation group is arbitrary, so the printed name is non-deterministic.
2. Task-dependency rows carry different task IDs — expected 85674001/85674000/85674003 vs actual
   85674005/85674004/85674007 — because the reference-table transfer tasks were scheduled in a different order.

The expected file is already self-evidently non-deterministic: lines 403-414 (the block_writes section)
contain the colg1/colg2/colg3 DEBUG lines in a jumbled order, while lines 214-225 (force_logical) are
ordered. The baseline was captured from one arbitrary run.

The test was introduced by f743b35fc — "Parallelize Shard Rebalancing & Unlock Concurrent Logical Shard
Moves" (#7983). Parallelization made scheduling non-deterministic; the test still asserts a fixed order.

C — statement_cancel_error_message: this is the one that actually matters

-ERROR:  canceling statement due to statement timeout
+ERROR:  canceling statement due to user request

Failing block (src/test/regress/sql/statement_cancel_error_message.sql), with
citus.force_max_query_parallelization TO ON set earlier in the file:

BEGIN;
SET LOCAL statement_timeout = '1ms';
INSERT INTO t_unrelated SELECT i FROM generate_series(1,10) i;

This test file exists solely as the regression guard for #3954 ("wrong cancel message") — see the header
comment on lines 1-2 of the .sql. So this failure is that original bug intermittently resurfacing.

When the local statement_timeout fires, Citus cancels the remote connections; the worker's generic
canceling statement due to user request can win the race and be surfaced to the client instead of the
coordinator's correct timeout attribution. This is a genuine defect with user-visible impact (misleading error
message), not test noise — it should not be resolved by loosening the expected output.

D / E — not Citus at all

D (#8749, arbitrary-configs-1) died ~50 s into the job, in the "Install python test env" step, before any
Citus test ran:

fatal: unable to access 'https://github.com/citusdata/mitmproxy.git/': server certificate verification failed. CAfile: none CRLfile: none
fatal: could not fetch c9ddda7d... from promisor remote
× git clone --filter=blob:none --quiet https://github.com/citusdata/mitmproxy.git ... did not run successfully. exit code: 128
ERROR: Couldn't install package: {}

No artifacts were uploaded. Transient network/TLS flake.

E (#8769, arbitrary-configs-3) — the check-run annotation reads:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner
process, starves it for CPU/Memory, or blocks its network access can cause this error.

Supporting evidence: step 5 check-arbitrary-configs (shard 3/6) has conclusion: null (never completed); the
job ran 46 minutes (03:45:30 → 04:31:31) while all 17 sibling shards finished in 2-8 minutes; no
cassert_17_arbitrary_configs_3 artifact was produced. Not a timeout-minutes expiry (that is 120). Most
likely OOM. Logs have since been purged.

Neither D nor E indicates a Citus problem. Auto-retry would be more appropriate than filing an issue.

---

Meta-finding: the issue count was inflated by the dedup scheme

The notify job in .github/workflows/nightly_cassert.yml dedups on a set-exact signature of failing
job names:

<!-- cassert-failures: ["cassert PG18 - regress"] -->

It appends to an existing open issue only when the failing job-name set matches exactly; any other
combination opens a brand-new issue. So the single recurring single_node_enterprise flake filed three
separate issues purely because its companions differed each night:

• Nightly cassert failures (2026-08-12) #8749 → ["cassert PG17 - arbitrary-configs-1", "cassert PG17 - regress"]
• Nightly cassert failures (2026-08-16) #8775 → ["cassert PG16 - regress", "cassert PG18 - regress"]
• Nightly cassert failures 2026-08-12..17 - consolidated triage (5 issues, 5 root causes, 0 assertions) #8776 → ["cassert PG18 - regress"]

Deduping on failing test names (or matching on set overlap) would have collapsed A into one issue and made
this triage unnecessary.

Separately, the "Publish regression.diffs" step contains an unquoted echo -E $diffs. Diff content contains
*, which the shell glob-expands against the repo root, injecting a stray directory listing
(CHANGELOG.md CODE_OF_CONDUCT.md ...) into the published diff — visible in the body of #8768, and actively
misleading during triage.

---

Actions in flight

Work has been started in parallel on the four actionable items:

1. C — statement_cancel_error_message — root-cause and fix the cancel-reason race (real bug, highest priority).
2. B — background_rebalance_parallel_reference_tables — make the test deterministic (re-baselining to another
   arbitrary run is explicitly not an acceptable fix).
3. A — single_node_enterprise — root-cause the extra connection; prefer pinning the test's executor settings
   over changing executor behaviour, unless the executor is genuinely wrong.
4. Workflow — fix the dedup signature and the unquoted echo -E $diffs.

D and E need no code action.

This issue stays open as the umbrella tracker until A, B and C are resolved.

[ihalatci]

Follow-up: investigation closed — 5 root causes, 2 are production defects

Following up on the consolidated analysis above. All five root causes have now been driven to a
conclusion. The headline has changed materially since the first comment: this was not five test
flakes. Two are real product defects, and a third closes a community-reported flake that has been
open for two years.

All five failing nightly runs were on the same commit, 52663ed4346d293f9561c1ca7bf3533cd6a11d9d
on main, so the night-to-night variation was pure non-determinism, not code drift. No assertion
ever fired in any of the seven failing jobs — every red job was an output diff or an infra flake.

This is also not a one-week cluster. The same three regress jobs (PG16, PG17, PG18) appear in
earlier nightly issues going back to at least 2026-07-17 — #8682, #8683, #8724. Logs for those runs
have since expired, so the specific failing tests cannot be confirmed and they are not counted as
evidence here; but combined with #7671 being open since 2024 and #8348 being open since 2025, the
pattern is chronic rather than recent.

Summary

	Failing job	Root cause	Class	Outcome
A	PG17/PG18 regress	single_node_enterprise — test asserts on a connection count that is racy by design	Test over-specification	Fixed. Closes #7671, addresses #8348 item 2
B	PG16 regress	background_rebalance_parallel_reference_tables — rebalancer picks an arbitrary colocation-group representative	Product defect	Fixed, causally reproduced on demand
C	PG16 regress	statement_cancel_error_message — distributed deadlock detector can SIGINT the wrong transaction	Product defect	Fixed (2 of 3 mechanisms); 1 mechanism documented, open
D	PG17 arbitrary-configs-1	mitmproxy git clone TLS verification failure	CI infra	No code action
E	PG17 arbitrary-configs-3	Hosted runner lost communication (likely OOM)	CI infra	No code action

Separately, the nightly workflow's issue de-duplication has been fixed so this stops spawning a new
issue per night.

All four fixes are up as draft PRs:

	PR	Branch
A	#8788 — Fix flaky single_node_enterprise multi-shard connection race	ihalatci-single-node-extra-connection
B	#8789 — Make rebalance colocation-group representative selection deterministic	ihalatci-rebalance-test-determinism
C	#8790 — Only cancel the distributed transaction that was found deadlocked	ihalatci-statement-cancel-error-message
D	#8791 — Fix nightly cassert issue dedup and regression.diffs glob expansion	ihalatci-cassert-workflow-issue-dedup

They are deliberately split: A is a test fix, B and C are independent product fixes in unrelated
subsystems, and D touches only CI. None of them shares a file with another.

---

A — single_node_enterprise: the test over-specifies a racy value

Not a product bug. CalculateNewConnectionCount() (adaptive_executor.c:2734) deterministically
decides to open 3 extra connections on every run, including every green one:
maxNewConnectionCount = targetPoolSize - initiatedConnectionCount (:2772),
newConnectionsForReadyTasks = Max(0, readyTaskCount - usableConnectionCount) (:2781), then
Min(4, 3) = 3 (:2794). The slow-start cap at :2784 is inside
if (ExecutorSlowStartInterval != SLOW_START_DISABLED) and the harness sets 0ms
(pg_regress_multi.pl:641), so it is skipped. The cost guard at :2843 cannot fire on the first cycle.

The determinism is in the decision. The race is only whether one of those connections completes TCP
connect and authentication before the six tiny sequential queries in the block drain. Under external
load it sometimes does, and its BEGIN appears in the log.

The test asserted on the count, so it recorded a value that was never guaranteed. The fix pins the
pool size for that block only.

Worth flagging for reviewers: the failure diff is byte-identical across every occurrence, which
looks like determinism but is the opposite. With exactly one last task, a winning connection has
exactly one place to insert its BEGIN — so a byte-identical diff is the signature this race
produces, not evidence against one.

Scheduling rules out internal interference: the test occupies its own test: line in both
enterprise_minimal_schedule (line 1) and enterprise_schedule (line 3), and no parallel-schedule
variant exists. Only external load can win the race.

This is not new. Two independent pre-existing reports match byte-for-byte at the identical hunk
header @@ -465,28 +465,30 @@:

• Flaky single_node_enterprise test #7671 — open since 2024-08-15
• Intermittent failures of unstable citus regression tests #8348 item 2 (unstable test list) — open since 2025-11-19

Incidentally #8348 item 8 attributes a separate failure to the OOM killer, which independently
corroborates root cause E below.

Fix: 2 files, +6/−0. The zero-deletion count is the verification: if pinning the pool had altered
execution, the existing NOTICE: issuing ... lines would have had to change. They did not.

B — rebalancer picks an arbitrary representative per colocation group

This one was initially misfiled as a test flake and is a genuine product defect.

NonColocatedDistRelationIdList() (shard_rebalancer.c:2013-2058) de-duplicates by colocationId,
keeping the first table it encounters (continue at :2050). It iterates
CitusTableTypeIdList(), which is an unordered systable_beginscan over pg_dist_partition
(metadata_cache.c:5078-5080).

So the representative of each colocation group — and therefore the shard IDs in the emitted rebalance
plan, and the table named in the DEBUG1: skipping child tables for relation named: ... line — is
whatever physical heap order happens to be. Any pg_dist_partition churn (HOT update, autovacuum,
page reuse) can flip it. That is exactly what the nightly hit.

The sibling test background_rebalance_parallel is equally exposed; it is only shielded from the
visible symptom because it never raises client_min_messages to DEBUG1. That ruled out a
test-only fix.

Reproduced on demand rather than waited for. Two arms on a fresh 3-node cluster with a forced HOT
update on pg_dist_partition, differing only in the library:

Library	representative before churn	after churn
Unpatched	table1_colg1	table2_colg1 ← bug reproduces
Patched	table1_colg1	table1_colg1 ← fixed

Measured OIDs were 17320 table1_colg1 < 17326 table2_colg1, so lowest-OID equals creation order,
which is what the committed baselines already assert. The fix converts an incidental property into a
guaranteed one.

Fix: 1 file, +7/−0 — sort by OID before the de-duplication, with a comment explaining why. Full
operations_schedule passes with zero expected-output churn; no baseline was re-recorded. A
six-call-site audit of the consumers came back clean.

C — deadlock detector can cancel the wrong transaction

The original hypothesis was wrong and is worth recording so nobody re-derives it. We assumed the
worker's generic cancellation won a race against the coordinator's deadlock-specific message. It
cannot have: the failing error carries no CONTEXT: while executing command on <host>:<port>
line, and at remote_commands.c:319-325 the errcontext(...) is an unconditional argument to the
same ereport (only messageContext is ternary-guarded at :323). Its absence proves the error was
raised locally on the coordinator.

The real defect is in the detector. Three distinct mechanisms were separated:

• (a) TOCTOU — the victim's distributed transaction changes between association and kill().
  The old guard only checked transactionNumber != 0. Fixed.
• (b) repeat SIGINT for a still-unwinding transaction — the daemon re-detects the same deadlock
  and signals again. Reproduced, not fixed, documented. See below.
• (c) cross-node transaction-number collision — nextTransactionNumber is initialised to 1
  per node (backend_data.c:564), and association matches on transactionNumber alone
  (distributed_deadlock_detection.c:398-402), never on the initiator. The Assert at :411 checks
  the live backend's initiator (bound at :396-397) and is structurally incapable of catching a
  graph-node mismatch. A node-1 transaction can therefore be matched to a same-numbered local
  transaction and SIGINT an innocent backend. Fixed — pre-existing, and not demonstrated for this
  specific test.

Fix: 3 files, +28/−5 — the cancel path now verifies both the transaction number and the
initiating node, i.e. it enforces at cancel time the invariant that association should already
guarantee. It only rejects associations that were already wrong, and nothing is orphaned: the DFS
only starts from the node's own transactions (:146), so a foreign node's deadlock is resolved by
that node's own detector.

Mechanism (b) is reproduced and left open. Under an amplified detector, 14 of 20 iterations issued
two SIGINTs to the same backend for the same transaction ~1 ms apart (34 cancels observed, all
matching). It is a real, common, pre-existing behaviour. The naive fix — skip when the
cancelledDueToDeadlock flag is set — is incomplete, because the flag is cleared on read by the
error-message hook. It is tracked separately rather than bundled here.

Why the failure is intermittent — the one thing we can state precisely

PostgreSQL 16 src/backend/tcop/postgres.c (verified against REL_16_STABLE) constrains the artifact
tightly. Inside else if (QueryCancelPending) at :3360:

• :3365 sets QueryCancelPending = false unconditionally, before any branch is chosen.
• :3391-3397 raises "canceling statement due to statement timeout" if the STATEMENT_TIMEOUT
  indicator is set — reached before the generic branch.
• :3421 guards the generic "canceling statement due to user request" with !DoingCommandRead,
  commented "If we are reading a command from the client, just ignore the cancel request."

Therefore the observed error requires a SIGINT delivered while the second block's INSERT is actually
executing, and before its statement_timeout = '1ms' indicator fires. That roughly-1-millisecond
window is the entire failure, and it is why the test passes on almost every run.

The same code refutes two theories we held, and we are recording both so they are not re-derived:

1. A pending cancel cannot leak across a statement boundary. :3365 consumes it and :3421
   discards it when the backend is between statements. This also explains mechanistically why
   mechanism (b) fired 14 times in the reproduction and converted zero times — the repeat SIGINT
   lands during the first block's unwind and is swallowed.
2. Widening deadlock_timeout cannot expose the mis-cancellation. A timing sweep over
   100 ms / 200 ms / 500 ms / 1 s produced 60 cancels, 60 correctly-matched, 0 mismatched, with the
   test passing at every setting. That null is not a small sample — it is structural:
   CheckForDistributedDeadlocks only reaches the kill() at distributed_deadlock_detection.c:211
   for a node that association already matched at :183 (non-matches continue at :190), and
   association matches on transaction-number equality. A correct match is therefore true by
   construction, and the only route to a mismatch is the narrow TOCTOU window between :183 and
   :211 — which deadlock_timeout does not widen, since it controls the interval between detector
   cycles. Anyone re-running this should not expect the mismatch counter to move.

Honest framing, which we are not going to overstate: this commit closes two real mis-cancellation
holes, (a) and (c). It is not proven to be the cause of the PG16 nightly failure, and both routes
we proposed from this fix to the artifact have now been refuted — one by the shape of the detector
code, one by PostgreSQL source. The delivery mechanism for the stray SIGINT remains unidentified; all
we can state is the timing requirement above.

What does justify shipping it independently of causation is that the change is monotonic: it only
ever adds a condition that can suppress a cancellation, and never adds one. The rate of stray
SIGINTs under the patch is therefore less than or equal to the rate without it, so it cannot make this
failure more likely, and no deadlock is orphaned — the DFS starts only from the node's own
transactions (:146), so a foreign node's cycle is resolved by that node's own detector, with
re-detection on the next cycle as the backstop.

D and E — CI infrastructure, no code action

• D (arbitrary-configs-1, 08-12): server certificate verification failed on
  git clone https://github.com/citusdata/mitmproxy.git, exit 128. The suite never ran.
• E (arbitrary-configs-3, 08-14): "The hosted runner lost communication with the server."
  46 minutes against a 2–8 minute norm, well under the 120-minute timeout-minutes, so not a
  timeout. Most likely OOM — corroborated by Intermittent failures of unstable citus regression tests #8348 item 8.

Neither is actionable in this repository. They are listed so they are not mistaken for unfinished work.

Nightly workflow — issue de-duplication

The de-duplication keyed on the exact sorted set of failing job names, which are PG-major
qualified. Two nights failing the same test on different PG majors produced disjoint keys, so one
flake opened multiple issues — which is why this investigation started with five issues instead of
one.

Changed to a single rolling open issue per label, selecting the newest matching issue and filtering
out pull requests. A shell-glob bug in the log-collection action (echo -E $diffs unquoted) was
fixed at the same time. +21/−40.

Practical consequence: subsequent nightly failures will append here rather than opening a sixth issue.