test: pin the guarantees that were only being argued for

birdnamoo · birdnamoo · commit 1c4f032fcf77 · 2026-07-19T18:41:51.000+09:00
CI gained a live job, and measuring what the suite actually catches turned
up more than expected.

Live tests were gated behind one flag covering two suites with different
needs: the self-contained Postgres one and the MorphDB one, which has no
redis or seeded tenant and so waits out Testcontainers' one-hour default.
Enabling either meant enabling both. Split the gate per suite, bound the
MorphDB readiness wait so it fails instead of stalling, and add a
postgres-live CI job now that the Postgres suite can run alone.

Then, measuring detection power by removing each guarantee and checking
for red: the append serialization and the deterministic-paging tie-break
were both fully uncovered, as were schema-name validation and the
projection snapshot bound. The concurrency test only asserts final state,
which a Postgres sequence satisfies on its own; the paging and snapshot
guards are invisible to the in-memory store because its LINQ sort is
stable and its stream copies the log before yielding. Each is now pinned
by a test that goes red when the guarantee goes away.

One guarantee stays uncovered on purpose: that watermark assignment order
equals commit order lives in a transient window the public API cannot hold
open, so a test for it would be a poller that can fail while the code is
correct. Said so in the code instead.

No production code changed.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -44,11 +44,47 @@ jobs:
       - name: Build
         run: dotnet build Formbase.slnx --no-restore --configuration Release
 
-      # Live/ tests (Testcontainers + real MorphDB) are excluded from compilation unless
-      # IncludeLiveTests=true, so the default suite needs no Docker host or service containers.
+      # Live tests are excluded from compilation unless their opt-in flag is set, so the default
+      # suite needs no Docker host or service containers. The Postgres suite runs in its own job.
       - name: Test
         run: dotnet test Formbase.slnx --no-build --configuration Release --verbosity normal
 
+  postgres-live:
+    name: Postgres Live Contract
+    runs-on: ubuntu-latest
+    # Gated on the cheap suite: a compile error or a plain-suite regression should not also burn
+    # container-pulling minutes here. The live flag changes which files compile, so this job cannot
+    # reuse the build job's output and rebuilds regardless.
+    needs: build
+    # The runner already provides a Docker daemon, and the fixture starts its own postgres through
+    # Testcontainers — so this needs no `services:` block, only the opt-in flag.
+    # Bounds the job independently of the fixture's own readiness timeout: a wedged daemon or a slow
+    # image pull costs a few minutes of Actions time, not the six-hour default.
+    timeout-minutes: 15
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup .NET
+        uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: ${{ env.DOTNET_VERSION }}
+
+      - name: Cache NuGet packages
+        uses: actions/cache@v4
+        with:
+          path: ~/.nuget/packages
+          key: ${{ runner.os }}-nuget-${{ hashFiles('**/Directory.Packages.props') }}
+          restore-keys: |
+            ${{ runner.os }}-nuget-
+
+      # The flag changes which files compile, so restore/build/test all carry it.
+      # MorphDB live stays off: its service needs redis and a seeded tenant that this harness
+      # does not yet provide, so enabling it here would fail the job by design.
+      - name: Test (Postgres live contract)
+        run: dotnet test Formbase.slnx --configuration Release --verbosity normal -p:IncludePostgresLiveTests=true
+
   quality:
     name: Format & Vulnerability Scan
     runs-on: ubuntu-latest
diff --git a/README.md b/README.md
@@ -116,17 +116,15 @@ dotnet build Formbase.slnx
 dotnet test  Formbase.slnx          # default suite — no Docker required
 ```
 
-Live tests stand up real backing services via Testcontainers and are excluded from the default build. Run them explicitly on a machine with Docker:
+Live tests stand up real backing services via Testcontainers and are excluded from the default build. The two suites need different things, so each has its own switch:
 
 ```bash
-dotnet test Formbase.slnx -p:IncludeLiveTests=true
+dotnet test Formbase.slnx -p:IncludePostgresLiveTests=true   # Docker only — self-contained
+dotnet test Formbase.slnx -p:IncludeMorphDbLiveTests=true    # also needs Redis + a seeded tenant
+dotnet test Formbase.slnx -p:IncludeLiveTests=true           # umbrella: both
 ```
 
-The PostgreSQL raw-store live tests run against a plain `postgres` container and pass out of the box. The MorphDB live tests additionally require a Redis service and a provisioned tenant, so they are best run in CI where those can be declared as service containers — filter to just the Postgres suite when running locally:
-
-```bash
-dotnet test Formbase.slnx -p:IncludeLiveTests=true --filter "FullyQualifiedName~PostgresRawStoreLiveContractTests"
-```
+The PostgreSQL raw-store live tests run against a plain `postgres` container and pass out of the box, which is why they are the ones worth enabling locally. The MorphDB suite is not runnable yet: MorphDB reports `/health` as 503 while Redis is absent, so its fixture times out (after two minutes — bounded deliberately, so an unreachable service fails rather than stalls). Completing that harness is tracked as its own roadmap phase.
 
 ## Roadmap
 
diff --git a/tests/Formbase.Core.Tests/Contracts/RawStoreContractTests.cs b/tests/Formbase.Core.Tests/Contracts/RawStoreContractTests.cs
@@ -142,9 +142,12 @@ public async Task Concurrent_appends_get_distinct_monotonic_watermarks()
         var store = CreateStore();
         const int count = 20;
 
-        // Launch all appends at once. For a store that assigns watermarks with a naive max()+1 or that
-        // lets assignment order diverge from commit order, this collides or drops rows; a correctly
-        // serialized store gives every append a distinct watermark and loses nothing.
+        // Launch all appends at once: a store that assigns watermarks with a naive max()+1 collides or
+        // drops rows, while a correct one gives every append its own watermark and loses nothing.
+        // This asserts on final state only, so it says nothing about *how* a store gets there — a
+        // Postgres sequence is concurrency-safe by itself, and this test stays green even with that
+        // adapter's append serialization removed (measured, cycle 21). What serialization buys is
+        // pinned per-adapter instead; see PostgresAppendSerializationTests.
         var appends = Enumerable.Range(0, count)
             .Select(i => store.AppendAsync(Qc, DocumentId.New(), Body($$"""{"n":{{i}}}""")))
             .ToArray();
diff --git a/tests/Formbase.Core.Tests/Formbase.Core.Tests.csproj b/tests/Formbase.Core.Tests/Formbase.Core.Tests.csproj
@@ -9,12 +9,31 @@
   </PropertyGroup>
 
   <!--
-    Live tests (Live/) stand up a real MorphDB container via Testcontainers and need a healthy Docker
-    host. They are excluded from compilation by default so the suite never depends on Docker and cannot
-    hang. Opt in with: dotnet test -p:IncludeLiveTests=true  (requires Docker).
+    Live tests stand up real services via Testcontainers and need a healthy Docker host, so they are
+    excluded from compilation by default: the suite never depends on Docker and cannot hang.
+
+    The two live suites have different requirements and are therefore gated independently — folding
+    them into one switch means the self-contained Postgres suite cannot run without also dragging in
+    the MorphDB one, which is not yet runnable (see below).
+
+      dotnet test -p:IncludePostgresLiveTests=true   # needs Docker only — self-contained
+      dotnet test -p:IncludeMorphDbLiveTests=true    # additionally needs redis + a seeded tenant
+      dotnet test -p:IncludeLiveTests=true           # umbrella: both
+
+    MorphDB live is expected to fail until its harness is completed (its /health stays 503 without
+    redis); it fails on a bounded timeout rather than hanging.
   -->
-  <ItemGroup Condition="'$(IncludeLiveTests)' != 'true'">
-    <Compile Remove="Live/**/*.cs" />
+  <PropertyGroup Condition="'$(IncludeLiveTests)' == 'true'">
+    <IncludePostgresLiveTests>true</IncludePostgresLiveTests>
+    <IncludeMorphDbLiveTests>true</IncludeMorphDbLiveTests>
+  </PropertyGroup>
+
+  <ItemGroup Condition="'$(IncludePostgresLiveTests)' != 'true'">
+    <Compile Remove="Live/Postgres/**/*.cs" />
+  </ItemGroup>
+
+  <ItemGroup Condition="'$(IncludeMorphDbLiveTests)' != 'true'">
+    <Compile Remove="Live/MorphDb/**/*.cs" />
   </ItemGroup>
 
   <ItemGroup>
diff --git a/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbEngineLiveTests.cs b/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbEngineLiveTests.cs
@@ -7,15 +7,15 @@
 using Formbase.Core.Schema;
 using Formbase.MorphDb;
 
-namespace Formbase.Core.Tests.Live;
+namespace Formbase.Core.Tests.Live.MorphDb;
 
 /// <summary>
 /// The flagship end-to-end scenario against a real MorphDB projection store: raw-first intake,
 /// then structure, then queryable records. Everything but the projection store is in-process;
 /// the projection lands in a real database. Requires Docker (category: Live).
 /// </summary>
 [Collection(MorphDbCollection.Name)]
-[Trait("Category", "Live")]
+[Trait("Category", "Live.MorphDb")]
 public sealed class MorphDbEngineLiveTests
 {
     private readonly MorphDbFixture _fixture;
diff --git a/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbFixture.cs b/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbFixture.cs
@@ -4,16 +4,25 @@
 using MorphDB.Client;
 using Testcontainers.PostgreSql;
 
-namespace Formbase.Core.Tests.Live;
+namespace Formbase.Core.Tests.Live.MorphDb;
 
 /// <summary>
 /// Spins up a real MorphDB service backed by PostgreSQL, both on a shared Docker network, once for
 /// all live tests. MorphDB ensures its own schema/extensions at startup, so a plain postgres suffices.
+/// <para>
+/// Incomplete on purpose: MorphDB's <c>/health</c> reports 503 while its optional redis dependency is
+/// absent, so readiness is never reached with this harness alone (a redis service and a seeded tenant
+/// are still missing). The readiness wait is therefore bounded — Testcontainers otherwise retries for
+/// its one-hour default, which stalls a CI job rather than failing it.
+/// </para>
 /// </summary>
 public sealed class MorphDbFixture : IAsyncLifetime
 {
     private const string PostgresAlias = "postgres";
 
+    /// <summary>Bounds the readiness wait so an unreachable service fails the run instead of stalling it.</summary>
+    private static readonly TimeSpan ReadinessTimeout = TimeSpan.FromMinutes(2);
+
     private readonly INetwork _network = new NetworkBuilder().Build();
     private PostgreSqlContainer _postgres = null!;
     private IContainer _morphdb = null!;
@@ -42,7 +51,7 @@ public async Task InitializeAsync()
             .WithEnvironment("ASPNETCORE_ENVIRONMENT", "Production")
             .WithPortBinding(8080, assignRandomHostPort: true)
             .WithWaitStrategy(Wait.ForUnixContainer()
-                .UntilHttpRequestIsSucceeded(r => r.ForPort(8080).ForPath("/health")))
+                .UntilHttpRequestIsSucceeded(r => r.ForPort(8080).ForPath("/health"), o => o.WithTimeout(ReadinessTimeout)))
             .Build();
         await _morphdb.StartAsync();
 
diff --git a/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbProjectionStoreLiveContractTests.cs b/tests/Formbase.Core.Tests/Live/MorphDb/MorphDbProjectionStoreLiveContractTests.cs
@@ -2,14 +2,14 @@
 using Formbase.Core.Tests.Contracts;
 using Formbase.MorphDb;
 
-namespace Formbase.Core.Tests.Live;
+namespace Formbase.Core.Tests.Live.MorphDb;
 
 /// <summary>
 /// Runs the projection-store contract against a real MorphDB service. Proves the adapter honors the
 /// same guarantees as the in-memory store. Requires Docker (category: Live).
 /// </summary>
 [Collection(MorphDbCollection.Name)]
-[Trait("Category", "Live")]
+[Trait("Category", "Live.MorphDb")]
 public sealed class MorphDbProjectionStoreLiveContractTests : ProjectionStoreContractTests
 {
     private readonly MorphDbFixture _fixture;
diff --git a/tests/Formbase.Core.Tests/Live/Postgres/PostgresAppendSerializationTests.cs b/tests/Formbase.Core.Tests/Live/Postgres/PostgresAppendSerializationTests.cs
@@ -0,0 +1,71 @@
+using Formbase.Core.Primitives;
+using Npgsql;
+
+namespace Formbase.Core.Tests.Live.Postgres;
+
+/// <summary>
+/// Pins what the append-path advisory lock actually buys, which the shared contract suite cannot see.
+/// <para>
+/// The contract suite's concurrency test asserts on <i>final</i> state — distinct watermarks, nothing
+/// lost. Postgres sequences are concurrency-safe on their own, so that assertion holds with or without
+/// the lock: removing the lock leaves the whole suite green (measured, cycle 21). What the lock is
+/// really for is the <b>check-then-insert window</b>: two transactions both read "no such id", both
+/// insert, and one dies on the primary key instead of returning idempotently. Holding the lock through
+/// commit closes that window — and because the store reads at READ COMMITTED, the waiter's lookup runs
+/// against a snapshot taken after the winner committed, so it sees the row and returns it.
+/// </para>
+/// <para>
+/// <b>Not covered here.</b> The lock also makes watermark <i>assignment order equal commit order</i>,
+/// without which a projection reading the head mid-flight could record a head above a document that
+/// commits later and skip it forever. That failure exists only inside a transient window, and the
+/// public API offers no way to hold a commit open, so it has no deterministic test — asserting it
+/// would mean a probabilistic poller that can fail while the code is correct. It stays covered by
+/// argument (see the type's remarks), not by test.
+/// </para>
+/// Requires Docker (category: Live.Postgres).
+/// </summary>
+[Collection(PostgresCollection.Name)]
+[Trait("Category", "Live.Postgres")]
+public sealed class PostgresAppendSerializationTests
+{
+    private const int Racers = 8;
+
+    private readonly PostgresFixture _fixture;
+
+    public PostgresAppendSerializationTests(PostgresFixture fixture) => _fixture = fixture;
+
+    [Fact]
+    public async Task Concurrent_appends_of_one_id_all_return_the_same_single_row()
+    {
+        var schema = "fb_append_" + Guid.NewGuid().ToString("N");
+        await using var racers = new RacingStores(_fixture, schema, Racers);
+        var type = FormTypeRef.Create("invoice");
+        var id = DocumentId.New();
+
+        // Every racer submits the same document id at once — the re-submission a retrying client or a
+        // second instance makes. Idempotency must hold under a race, not just in sequence.
+        var stored = await racers.RaceAsync(store => store.AppendAsync(type, id, DocumentBody.Parse("""{"n":1}""")));
+
+        stored.Select(document => document.Watermark.Value).Distinct()
+            .Should().ContainSingle("every racer must observe the one document that won, not its own insert");
+        (await CountRowsAsync(schema, id)).Should().Be(1, "a duplicate id must never create a second row");
+        (await SequenceValueAsync(schema)).Should().Be(1, "the losing racers must not burn watermarks");
+    }
+
+    private async Task<int> CountRowsAsync(string schema, DocumentId id)
+    {
+        await using var connection = await _fixture.DataSource.OpenConnectionAsync();
+        await using var command = new NpgsqlCommand(
+            $"""SELECT count(*) FROM "{schema}".raw_documents WHERE id = @id""", connection);
+        command.Parameters.AddWithValue("id", id.Value);
+        return (int)(long)(await command.ExecuteScalarAsync())!;
+    }
+
+    private async Task<long> SequenceValueAsync(string schema)
+    {
+        await using var connection = await _fixture.DataSource.OpenConnectionAsync();
+        await using var command = new NpgsqlCommand(
+            $"""SELECT last_value FROM "{schema}".raw_watermark_seq""", connection);
+        return (long)(await command.ExecuteScalarAsync())!;
+    }
+}
diff --git a/tests/Formbase.Core.Tests/Live/Postgres/PostgresColdStartRaceTests.cs b/tests/Formbase.Core.Tests/Live/Postgres/PostgresColdStartRaceTests.cs
@@ -0,0 +1,69 @@
+using Formbase.Core.Primitives;
+using Formbase.Postgres;
+using Npgsql;
+
+namespace Formbase.Core.Tests.Live.Postgres;
+
+/// <summary>
+/// Pins the cold-start guarantee: many instances reaching a fresh schema at once must all succeed.
+/// <para>
+/// Each store gates its own initialization with a private <c>SemaphoreSlim</c>, which does nothing
+/// across instances (see <see cref="RacingStores"/>) — so the only thing standing between concurrent
+/// cold starts and a failure is the schema-scoped advisory lock around the DDL. Remove that lock and
+/// these tests fail: <c>CREATE ... IF NOT EXISTS</c> is not atomic against the catalog, so racing
+/// creators collide on a duplicate-object error (verified by mutation, cycle 20).
+/// </para>
+/// Requires Docker (category: Live.Postgres).
+/// </summary>
+[Collection(PostgresCollection.Name)]
+[Trait("Category", "Live.Postgres")]
+public sealed class PostgresColdStartRaceTests
+{
+    private const int Racers = 8;
+
+    private readonly PostgresFixture _fixture;
+
+    public PostgresColdStartRaceTests(PostgresFixture fixture) => _fixture = fixture;
+
+    [Fact]
+    public async Task Concurrent_cold_start_creates_the_schema_exactly_once()
+    {
+        var schema = NewSchema();
+        await using var racers = new RacingStores(_fixture, schema, Racers);
+
+        // A read is enough to trigger initialization, so this isolates the DDL race from the append path.
+        var heads = await racers.RaceAsync(store => store.HeadAsync(FormTypeRef.Create("invoice")));
+
+        heads.Should().AllSatisfy(head => head.Value.Should().Be(0), "an empty store has no watermark yet");
+        (await CountTablesAsync(schema)).Should().Be(1, "the racing creators must converge on one table");
+    }
+
+    [Fact]
+    public async Task Concurrent_cold_start_with_appends_assigns_unique_watermarks()
+    {
+        var schema = NewSchema();
+        await using var racers = new RacingStores(_fixture, schema, Racers);
+        var type = FormTypeRef.Create("invoice");
+
+        // Initialization and the first append collide in the same window — the DDL lock and the append
+        // lock are the same key, so this also proves the two paths do not deadlock against each other.
+        var appended = await racers.RaceAsync(store =>
+            store.AppendAsync(type, DocumentId.New(), DocumentBody.Parse("""{"n":1}""")));
+
+        appended.Select(document => document.Watermark.Value)
+            .Should().BeEquivalentTo(Enumerable.Range(1, Racers).Select(n => (long)n),
+                "each racer consumes exactly one watermark from the shared sequence");
+    }
+
+    private async Task<int> CountTablesAsync(string schema)
+    {
+        await using var connection = await _fixture.DataSource.OpenConnectionAsync();
+        await using var command = new NpgsqlCommand(
+            "SELECT count(*) FROM information_schema.tables WHERE table_schema = @s AND table_name = 'raw_documents'",
+            connection);
+        command.Parameters.AddWithValue("s", schema);
+        return (int)(long)(await command.ExecuteScalarAsync())!;
+    }
+
+    private static string NewSchema() => "fb_race_" + Guid.NewGuid().ToString("N");
+}
diff --git a/tests/Formbase.Core.Tests/Live/Postgres/PostgresFixture.cs b/tests/Formbase.Core.Tests/Live/Postgres/PostgresFixture.cs
@@ -1,7 +1,7 @@
 using Npgsql;
 using Testcontainers.PostgreSql;
 
-namespace Formbase.Core.Tests.Live;
+namespace Formbase.Core.Tests.Live.Postgres;
 
 /// <summary>
 /// Spins up a plain PostgreSQL once for all raw-store live tests and exposes a shared data source. Each
@@ -14,6 +14,13 @@ public sealed class PostgresFixture : IAsyncLifetime
 
     public NpgsqlDataSource DataSource { get; private set; } = null!;
 
+    /// <summary>
+    /// Builds a data source with its own connection pool, independent of <see cref="DataSource"/>. Stores
+    /// built over separate pools share no in-process state, which is how a multi-process cold start is
+    /// approximated in a single test host. The caller disposes what it creates.
+    /// </summary>
+    public NpgsqlDataSource CreateIndependentDataSource() => NpgsqlDataSource.Create(_postgres.GetConnectionString());
+
     public async Task InitializeAsync()
     {
         _postgres = new PostgreSqlBuilder("postgres:16-alpine").Build();
diff --git a/tests/Formbase.Core.Tests/Live/Postgres/PostgresRawStoreLiveContractTests.cs b/tests/Formbase.Core.Tests/Live/Postgres/PostgresRawStoreLiveContractTests.cs
@@ -2,7 +2,7 @@
 using Formbase.Core.Tests.Contracts;
 using Formbase.Postgres;
 
-namespace Formbase.Core.Tests.Live;
+namespace Formbase.Core.Tests.Live.Postgres;
 
 /// <summary>
 /// Runs the raw-store contract against a real PostgreSQL. Proves the durable adapter honors the same
@@ -11,7 +11,7 @@ namespace Formbase.Core.Tests.Live;
 /// Requires Docker (category: Live).
 /// </summary>
 [Collection(PostgresCollection.Name)]
-[Trait("Category", "Live")]
+[Trait("Category", "Live.Postgres")]
 public sealed class PostgresRawStoreLiveContractTests : RawStoreContractTests
 {
     private readonly PostgresFixture _fixture;
diff --git a/tests/Formbase.Core.Tests/Live/Postgres/RacingStores.cs b/tests/Formbase.Core.Tests/Live/Postgres/RacingStores.cs
diff --git a/tests/Formbase.Core.Tests/Postgres/PostgresSchemaNameTests.cs b/tests/Formbase.Core.Tests/Postgres/PostgresSchemaNameTests.cs
diff --git a/tests/Formbase.Core.Tests/Projection/ProjectorTests.cs b/tests/Formbase.Core.Tests/Projection/ProjectorTests.cs
diff --git a/tests/Formbase.Core.Tests/Query/RecordQueryTests.cs b/tests/Formbase.Core.Tests/Query/RecordQueryTests.cs
