[Hackdays 40] Online Schema Migrations investigation and prototype - #22
Closed
driv3r wants to merge 31 commits into
Closed
[Hackdays 40] Online Schema Migrations investigation and prototype#22driv3r wants to merge 31 commits into
driv3r wants to merge 31 commits into
Conversation
…dmap Add a design roadmap for online (near-zero-downtime) YSQL schema changes that require a table rewrite today. The approach builds a hidden shadow DocDB generation, copies rows at a snapshot HybridTime, keeps it current via asynchronous WAL/CDC mirroring, and performs an atomic storage switch that preserves the source `pg_class.oid`. Covers: current DDL classification (metadata-only vs rewrite), shadow creation and distributed copy, WAL/CDC mirroring with idempotent apply, near-term relfilenode cutover and long-term logical-to-physical generation pointer, and CDC/backup/PITR/partition/geo/colocation behavior across migration phases. xCluster is explicitly out of scope. --- _automated · OpenCode (Claude Opus 4.8)_
…tion harness Throwaway validation glue (not production code) for the online schema change roadmap. Drives the end-to-end thin slice against a local cluster using only existing YSQL primitives: create shadow table, snapshot backfill, async mirror via logical replication slot (`test_decoding` + `pg_recvlogical`), apply with the target transform, cutover, and row-parity assertions. - `mirror_harness.sh`: 1:1 single tablet-set flow. - `mirror_harness_nm.sh`: N:M distributed flow (source 4 tablets, shadow 6), primary-key-changing transform, multi-row cross-tablet transactions; asserts target fan-out, transaction atomicity, and exact parity. - `apply_changes.py`: parses `test_decoding` output into shadow apply SQL, preserving source BEGIN/COMMIT framing into atomic target transactions. - `README.md`, `NEXT-STEPS.md`: results and first in-tree implementation plan. Both harnesses pass. Validates the distributed shape of the design: a single logical shadow table is a multi-tablet distributed table, source and target tablet layouts need not match, and rows route by the transformed primary key. --- _automated · OpenCode (Claude Opus 4.8)_
… Step 1) Pin the cutover mechanic the async shadow-table online schema change roadmap depends on: a table rewrite must preserve the relation's stable logical identity (`pg_class.oid`) while replacing its physical storage (`pg_class.relfilenode` -> a new DocDB table), and OID-bound dependents (views, foreign keys) must keep working afterwards. This is explicitly not a MySQL-style rename swap. `pg_online_schema_change-test`: - `SwapPreservesOid`: a volatile-default `ADD COLUMN` (forces a rewrite) keeps the same `pg_class.oid`, moves to a new `relfilenode`, and materializes the new column on all existing rows. - `SwapPreservesDependents`: after rewriting a parent table, a dependent view still resolves and a child foreign key still enforces. Both pass. Confirms the existing YB table-rewrite path already performs the OID-preserving storage switch; the remaining roadmap work is to drive that swap from an externally-built, caught-up shadow generation rather than an inline snapshot copy. See architecture/design/online-schema-changes-async-shadow-roadmap.md. --- _automated · OpenCode (Claude Opus 4.8)_
…edger (Step 2) Harden the shadow-mirror prototype from batch apply into a live streaming applier that survives an applier restart mid-stream with exactly-once effect. - `streaming_applier.py`: consumes test_decoding output and, for each SOURCE transaction (keyed by xid), emits ONE target transaction that applies the transformed mutations together with an idempotency-ledger row `(slot, xid)`. A DO-block early-return makes a replayed xid a full no-op (skips the mutations, not just the ledger insert). - `mirror_harness_restart.sh`: captures changes WITHOUT sending slot feedback and is killed mid-stream, so the slot replays everything on the next pass; asserts the ledger has no duplicate `(slot, xid)`, redelivery actually occurred, and final shadow/source parity is exact. Result: pass 1 (killed, no ack) applies 4 commits; pass 2 replays all 8; ledger ends at 8 rows with 0 duplicates; parity 2200 == 2200. This demonstrates exactly-once EFFECT on top of at-least-once transport. Keyed on source `xid` rather than commit LSN: the SQL query API that exposes per-commit LSN is gated behind a preview flag that did not take effect here; `(slot, xid)` is sufficient for the demonstration. See architecture/design/online-schema-changes-async-shadow-roadmap.md and prototype/osc-logical-slot/NEXT-STEPS.md. --- _automated · OpenCode (Claude Opus 4.8)_
…riter (Step 3) Replace the shadow-mirror prototype's fixed-sleep drain with a deterministic "mirror caught up to a barrier" check, validated under a continuous background writer. `mirror_harness_barrier.sh`: - A background writer mutates the source throughout (builds a real change tail). - To finalize, take ACCESS EXCLUSIVE on the source (fences new writes), insert a unique BARRIER sentinel row, then drain the slot and apply it, polling until the sentinel's transformed row is visible in the shadow. That visibility is the caught-up signal; no blind sleep. - After cutover, parity is exact including the full writer tail. Fixes found while building: - `streaming_applier.py`: the `new_id = id*2` transform now targets bigint and casts (`id::bigint*2`); doubling a large int source id overflowed int4 and aborted the batch. - Drain once after the barrier rather than looping many short-lived pg_recvlogical consumers, which collide on the slot lease and capture nothing. Cutover in the harness remains a name swap; the OID-preserving storage switch is validated separately in-tree by pg_online_schema_change-test (Step 1). Wiring the two together, plus a master-owned job and a per-tablet resolved-HybridTime barrier, is backend work tracked in NEXT-STEPS.md. See architecture/design/online-schema-changes-async-shadow-roadmap.md. --- _automated · OpenCode (Claude Opus 4.8)_
Update the async shadow-table roadmap with what the prototype validated so the document is self-contained and the next work is explicit. - Add "Prototype Findings (validated)": OID-preserving storage switch already exists; N:M distributed fan-out and cross-tablet transaction atomicity hold; exactly-once effect via a same-transaction idempotency ledger; deterministic sentinel-based barrier drain. Records env constraints (slot query-API preview flag, non-comparable LSN spaces, pg_recvlogical slot-lease behavior, bigint transform key) and what remains unproven. - Mark milestone statuses ([proto]/[done]/partial) and add a "Next Investigations" list: generation metadata + hidden shadow, wiring the caught-up shadow into the OID-preserving switch seam (tablecmds.c:6390-6409), master-owned job, per-tablet resolved-HT barrier, internal CDCSDK capture, external CDC handoff, backup/PITR, shapes. - Add a "Prototype Artifacts" section pointing at the in-tree test and prototype/osc-logical-slot/. --- _automated · OpenCode (Claude Opus 4.8)_
…o OSC roadmap Fold the migration status-tracking investigation into the async shadow-table roadmap. - Decisions/Scope: async submission returns a durable server-generated `migration_id` (snapshot-restoration contract, not the blocking `CREATE INDEX` contract); ordinary DDL wire behavior preserved (no UUID in the command tag or as a result row); status model is generic (`kind`) with OSC as first producer; status exposed as queryable relations; optional client idempotency token. - Findings: surveyed existing status surfaces (`BackfillJobPB`, `pg_stat_progress_create_index`, snapshot `restoration_id`, `yb_servers`/`yb_tablet_metadata` SRF-view predicate behavior) and current instant/existing DDL cost classes; recorded the driver-compatibility reasons (libpq `PGRES_TUPLES_OK` flip, pgjdbc trailing-token parse, JDBC `executeUpdate(DDL)`). - New Section 0: submission/identity, two-tier storage (`SysSchemaMigrationEntryPB` summary + internal per-work-unit progress table), queryable `yb_schema_migrations` / `yb_schema_migration_progress` relations with `migration_id` pushdown, progress/percentage semantics, failover reload. - Cross-reference Section 1.2 job metadata and Section 2.7 state machine to the public `state`/`phase`/`state_epoch` columns. - Roadmap: add Milestone A0 and Next Investigation #0 (durable async job + queryable status lands before copy/replay); add required tests and Open Decisions (status-relation pushdown, submission surface, retention, percentage semantics); extend Non-Goals. --- _automated · OpenCode (Claude Opus 4.8)_
First chunk of the online-schema-change async migration state tracker (Milestone A0, roadmap Section 0). Registers a durable, generic schema-migration job entity so a later manager can admit jobs, persist state, and reload after master failover. - Add `SCHEMA_MIGRATION = 22` to `SysRowEntryType` and the corresponding `CATALOG_ENTITY_TYPE_MAP` entry (append-only; static_assert kept intact). - Add `SysSchemaMigrationEntryPB` to `catalog_entity_info.proto`: `kind` (ONLINE_TABLE_REWRITE first), lifecycle `state`, execution `phase`, `state_epoch` for failover fencing, target/submitter OIDs, DDL text, timestamps, `terminal_error`, and an optional client `request_id` idempotency token. The migration id is the sys-catalog row key, mirroring `SysRestorationEntryPB`. - Handle the new type in the two exhaustive `SysRowEntryType` switches: `GenerateIdUnlocked` returns a unique id (uniqueness enforced by the manager's map, like SNAPSHOT/CLONE_STATE); `ImportSnapshotPreprocess` rejects it, since migration jobs are never part of a user backup. No behavior change yet; nothing creates or reads this entity. Upgrade-safe: enum values are appended and the sys.catalog physical schema is unchanged. --- _automated · OpenCode (Claude Opus 4.8)_
Second A0 chunk: the in-memory COW wrapper and sys-catalog loader for the schema-migration job entity, modeled on CloneStateInfo. - `SchemaMigrationInfo` (MetadataCowWrapper over `SysSchemaMigrationEntryPB`) keyed by the migration id (server-generated UUID string, the sys-catalog row key). `SchemaMigrationInfoHelpers::IsTerminal` centralizes the terminal-state check. - `DECLARE_MULTI_INSTANCE_LOADER_CLASS(SchemaMigration, ...)` so the manager can reload jobs from sys.catalog on leader change. - Forward declarations `SchemaMigrationManager` / `SchemaMigrationInfo` + `SchemaMigrationInfoPtr` in `master_fwd.h`. - Register `schema_migration/schema_migration_entity.cc` in the master build. Still no manager and no producers/consumers; this only makes the entity loadable. --- _automated · OpenCode (Claude Opus 4.8)_
…sion Third A0 chunk: the master-owned manager that admits jobs, persists every transition, retains terminal jobs, and reloads after leader failover. Modeled on CloneStateManager but resumes (rather than aborts) non-terminal jobs on failover. - `StartSchemaMigration` generates the server UUID, writes the initial NEW row, waits for replication, then returns the id (durable-before-return, matching snapshot creation). Supports `request_id` lost-response idempotency (a retried submission resolves to the existing job). Admission is gated behind `TEST_enable_schema_migration_admission`. - `GetSchemaMigration` / `ListSchemaMigrations` (optional state filter) / `CancelSchemaMigration` (persists CANCELLING and bumps `state_epoch` before signalling, so a failover cannot resume a cancelled job). - `ClearAndRunLoaders` + `LoadSchemaMigration`: reload from sys.catalog; bump `state_epoch` on non-terminal jobs to fence stale-epoch callbacks; keep terminal jobs for lookup. - `Run` is the skeleton executor: NEW -> RUNNING(PREFLIGHT) -> SUCCEEDED, and CANCELLING -> CANCELLED, with `TEST_pause_schema_migration_in_running` and `TEST_fail_schema_migration_in_running` hooks. It does NOT touch the target table; the real copy/replay/cutover phases wire in here later. - Own the manager on `Master`; run loaders from `RunLoaders`, post-load from `SysCatalogLoaded`, and the periodic tick from the catalog-manager bg loop (with the current LeaderEpoch). Still no external surface (RPC/SQL); exercised next via a unit test. --- _automated · OpenCode (Claude Opus 4.8)_
Fourth A0 chunk: the MasterAdmin RPC surface over SchemaMigrationManager. - master_admin.proto: `StartSchemaMigration` (returns server-generated `migration_id`), `GetSchemaMigration`, `ListSchemaMigrations` (optional state filter), `CancelSchemaMigration`, plus a `SchemaMigrationInfoPB` wrapper (id + `SysSchemaMigrationEntryPB` entry). Reuses the already-imported catalog_entity_info types. - master_admin_service.cc: implement the four handlers explicitly (like WriteSysCatalogEntry) rather than via the handler-macro, since the manager API is domain-typed. Start/Cancel run on the leader and pass `l.epoch()`; Get/List run on the leader (follower in-memory state is not maintained). Upgrade note: appending RPCs to an existing service; old masters simply lack the methods. Admission is still test-gated in the manager, so no behavior change on a default cluster. --- _automated · OpenCode (Claude Opus 4.8)_
Fifth A0 chunk: end-to-end coverage of the durable tracker on a 3-master mini-cluster, driving the manager directly (RPC surface comes next). - `AdmitAndSucceed`: admission returns a resolvable id immediately (durable-before-return); the skeleton executor reaches SUCCEEDED; the terminal job stays queryable. - `RequestIdIdempotency`: a duplicate `request_id` resolves to the same job; no second job is created. - `Cancel`: a paused RUNNING job cancels to CANCELLED; cancelling a terminal job fails. - `FailoverResumesRunningJob`: with the job paused in RUNNING, step down the master leader; the new leader reloads the job from sys.catalog under the same id, and once unpaused it completes exactly once (validates reload + state_epoch fencing). All four pass. Uses `TEST_enable_schema_migration_admission` and `TEST_pause_schema_migration_in_running`. --- _automated · OpenCode (Claude Opus 4.8)_
…te to C API Sixth A0 chunk: the full C++ transport from Postgres down to the master, for the schema-migration Start/Get/List/Cancel operations. - YBClient: `StartSchemaMigration` / `GetSchemaMigration` / `ListSchemaMigrations` / `CancelSchemaMigration` (pass-through to the MasterAdmin RPCs via CALL_SYNC_LEADER_MASTER_RPC_EX); add the Admin specializations. Include master_admin.fwd.h in client.h. - PgClientService: four handlers translating the flattened PG request/response PBs to/from the master PBs (`FillPgSchemaMigrationInfo` maps enum values to their string names). Register them in YB_PG_CLIENT_METHODS. - pg_client.proto: flattened Pg* request/response messages + PgSchemaMigrationInfoPB (scalar fields so pggate/PG need not depend on catalog_entity_info). - pggate PgClient/PgApiImpl forwarders + PggateRPC enum entries. - ybc_pggate C API: `YBCStartOnlineSchemaChange`, `YBCCancelSchemaMigration`, `YBCGetSchemaMigrations`, and `YbcPgSchemaMigrationInfo` (HybridTime -> Postgres-epoch conversion for timestamps). Compiles through yb-tserver; PG catalog functions/views wire in next. --- _automated · OpenCode (Claude Opus 4.8)_
Seventh A0 chunk: the user-facing SQL surface, verified end-to-end on a local cluster. - pg_yb_utils.c: `yb_start_online_schema_change(ddl text, request_id text)` (row-returning, returns the server-generated migration id; raises a clear error if the master does not admit the migration), `yb_cancel_schema_migration(migration_id text)`, and the `yb_get_schema_migrations(state_filter text)` SRF. Start/cancel enforce `IsYbDbAdminUser`. - pg_proc.dat: register the three functions (oids 8119-8121). Start and the SRF are `proisstrict => f` so a NULL request_id / NULL filter still executes. - yb_system_views.sql: `yb_schema_migrations` view over the SRF; query with an ordinary `WHERE migration_id = '...'`. - yb_system_functions.sql: REVOKE start/cancel from public, GRANT to `yb_db_admin`; the read view stays public (like yb_database_clones). Manual verification (RF1, TEST_enable_schema_migration_admission=true): start returns an id; `WHERE migration_id = ...` works; the skeleton executor advances NEW -> SUCCEEDED with terminal rows retained (including across a master restart); duplicate request_id is idempotent; cancel drives RUNNING -> CANCELLED; with the gate off, admission is refused with a clear error. Note: existing clusters need a `Vxxx` YSQL migration to add these catalog entries; that migration and a pg_schema_migration-test are the next chunk. Follow-up: surface the master's NotSupported status as a typed RPC error instead of relying on the empty-id check. --- _automated · OpenCode (Claude Opus 4.8)_
…el test Final A0 chunk: upgrade path for existing clusters plus SQL end-to-end tests. - V107__4192__yb_schema_migrations.sql: installs the three functions (oids 8119-8121), pg_depend pin records, yb_db_admin ACLs for start/cancel, and the yb_schema_migrations view for clusters initialized before these catalog entries existed. Column values (pronargs/proisstrict/proretset/prorettype/ proacl) verified to match a fresh initdb catalog. Bump the migration baseline in pg_yb_migration.dat to V107. - pg_schema_migration-test.cc (PgMiniTestBase): StartQueryAndSucceed (id resolvable, executor -> SUCCEEDED, kind recorded), RequestIdIdempotency, Cancel (RUNNING -> CANCELLED), AdmissionGate (disabled admission raises and admits nothing). All four pass. This completes Milestone A0: an online schema change can be started from SQL (returns a durable migration id), monitored via `SELECT ... FROM yb_schema_migrations WHERE migration_id = ...`, and cancelled, with jobs surviving master failover. The executor is still a skeleton that does not modify the target table; the real copy/replay/cutover phases wire into SchemaMigrationManager::AdvanceJob next. --- _automated · OpenCode (Claude Opus 4.8)_
A0 polish: complete the two-tier observability contract from roadmap Section
0.3. The summary (yb_schema_migrations) was already in place; this adds the
per-work-unit detail surface.
- New SRF yb_get_schema_migration_progress(state_filter) + view
yb_schema_migration_progress (oid 8122), columns: migration_id, work_kind,
source_table_id, tablet_id, state, rows_done, rows_total, updated_time.
- Until the distributed copy/replay backend produces real per-tablet/range
progress, it emits a single synthetic whole-job ('JOB') work unit per
migration, so the queryable contract (WHERE migration_id = ...) is exercised
end to end now and the shape is stable for consumers.
- Registered in pg_proc.dat, initdb views, and the V107 YSQL migration for
existing clusters.
- pg_schema_migration-test: new ProgressView test. All 5 SQL-level tests pass.
---
_automated · OpenCode (Claude Opus 4.8)_
…w tables Milestone A: represent a hidden second physical copy of a live table (a "shadow generation") for online schema changes, and keep it invisible to every user-facing enumeration. See architecture/design/online-schema-changes/ generation-metadata.md. - SysTablesEntryPB: add `PhysicalGenerationRole` enum (ACTIVE/SHADOW/RETIRED, field 43, default ACTIVE) and `owning_migration_id` (field 44), orthogonal to the existing HideState (which tracks dropped/hidden user objects). - PersistentTableInfo accessors: physical_generation_role(), is_active_generation()/is_shadow_generation()/is_retired_generation(); visible_to_client() now also requires the active generation, so most user-facing paths exclude shadows for free. - Explicit exclusions where client-visibility is bypassed: ListTables (skip non-active generations even with include_not_running), CDC discovery (IsTableEligibleForCDCSDKStream), and backup enumeration (via visible_to_client). Hidden-object GC is untouched because a shadow stays hide_state=VISIBLE. Upgrade-safe: fields appended, default ACTIVE == pre-feature behavior; no shadow is created until a (gated) online schema change runs. Shadow creation, adoption at cutover, and GC wire in with the migration job next. --- _automated · OpenCode (Claude Opus 4.8)_
Step 3 of the immediate execution order: pin the invisibility invariant that online-schema-change cutover relies on, using the physical-generation role from the previous commit. - pg_online_schema_change-test: new ShadowGenerationHiddenFromListTables. Create a YSQL table (visible in ListTables), flip its master TableInfo physical_generation_role to SHADOW, and assert it disappears from the client-facing ListTables view while its pg_class row remains; restoring ACTIVE makes it visible again. - Adds test helpers FindUserTable / CountInListTables over catalog_manager_impl(). This validates the enumeration-exclusion path end to end in a mini-cluster before the migration job starts creating real SHADOW generations. The existing SwapPreservesOid / SwapPreservesDependents tests still pass. --- _automated · OpenCode (Claude Opus 4.8)_
…se pipeline Step 4 of the immediate execution order: make the migration executor advance through an observable phase pipeline inside RUNNING, one phase per background tick, persisting each transition: NEW -> RUNNING[PREFLIGHT -> SHADOW_CREATING -> COPYING -> CUTOVER] -> SUCCEEDED These phases surface through yb_schema_migrations.phase and yb_schema_migration_progress, so consumers see a real lifecycle. No DocDB storage work is performed yet (no table is mutated) - the phases are the seams where the shadow-create / copy / replay / cutover backend plugs in. Cancel and the TEST failure hook still short-circuit to CANCELLED/FAILED. - schema_migration_manager.cc: phase constants + NextPhase(); RUNNING advances a phase per tick and only succeeds after CUTOVER. - schema_migration_manager-itest: new PhasePipeline test (job passes through the CUTOVER phase before SUCCEEDED). All manager and SQL-level tests still pass. - generation-metadata.md: document the phase pipeline and the per-phase backend seams (shadow create, adopt-at-cutover) that remain. Deferred to Step 5: real SHADOW generation creation (master CreateTable with role + owning migration id) and adopting a prefilled shadow into the source OID at CUTOVER, plus copy/replay/barrier. --- _automated · OpenCode (Claude Opus 4.8)_
…n job Real hidden-generation creation for online schema changes. The SHADOW_CREATING phase now builds an actual hidden second physical DocDB generation of the source table, entirely master-side (no Postgres backend, no client RPC). - CatalogManager::CreateShadowGeneration(source_table_id, migration_id, epoch): read-locks the source (require non-colocated, non-index, active YSQL table), copies its schema/partition_schema/tablet-count into a CreateTableRequestPB, sets pg_table_id to the source's logical id (shared identity), mints a fresh relfilenode from the source database's OID space via ReservePgsqlOids, sets the new physical table_id, and calls the internal CreateTable(rpc=nullptr). - CreateTableRequestPB gains physical_generation_role + owning_migration_id (fields 36/37); CreateTableInfo applies them, so the table is SHADOW from its first sys-catalog write and never transiently visible as ACTIVE. - SysSchemaMigrationEntryPB.shadow_table_id (field 14) records the result. - SchemaMigrationManager::PerformPhaseWork runs SHADOW_CREATING without holding the job lock (avoids lock inversion), is idempotent across failover (skips if shadow_table_id already set), and fails the job on error. Gated by TEST_schema_migration_create_shadow until the copy/replay/cutover backend lands, so master-only itests keep running the observable phase pipeline without a real source table. Tests (pg_online_schema_change-test): CreateShadowGeneration (direct) and MigrationCreatesShadowGeneration (end-to-end through the job) - shadow exists, is SHADOW, owned by the migration, shares the source logical id, distinct physical id, and stays out of ListTables. All manager itests still pass. Upgrade-safe: new proto fields are optional (default ACTIVE/empty); old masters ignore them. No shadow is created unless a gated online schema change runs. --- _automated · OpenCode (Claude Opus 4.8)_
…_oid from SQL Extend the online-schema-change job to run the real backend end to end (gated by TEST_schema_migration_create_shadow): SHADOW_CREATING -> COPYING -> CUTOVER. COPYING (CatalogManager::CopyGenerationData): bulk-copy source -> shadow by tablet clone (SST hard-link), reusing the clone infrastructure. Snapshots both tables at a fixed time and waits COMPLETE (CreateAndWaitTableSnapshot, collecting entries without kAddIndexes since single-YSQL-table index collection is unsupported), zips source/shadow tablets by partition start key (valid: same partition schema + tablet count), seeds shadow tablet consensus peers from the source, and issues one AsyncCloneTablet per pair. Prototype scope: quiesced source (no incremental replay yet), RF1, non-colocated single heap. CUTOVER (CatalogManager::CutoverToShadow): flip roles in one sys-catalog batch (shadow -> ACTIVE, source -> RETIRED). NOTE: this master-side flip does not redirect I/O by itself - YSQL resolves the physical table from pg_class.relfilenode per query. The load-bearing PG-layer relfilenode repoint is the remaining piece; the role flip is validated in isolation for now. Plumbing: yb_start_online_schema_change now takes (rel regclass, ddl, request_id) and passes the relation's relfilenode (YbGetRelfileNodeId, the YB physical id) as table_oid, so the master resolves the correct source generation. Updated the C API, pg_proc.dat, yb_system_functions grants, and the V107 migration. TEST_schema_migration_stop_before_cutover lets tests observe the populated shadow while still SHADOW. Tests (pg_online_schema_change-test): MigrationCreatesShadowGeneration (stops before cutover), MigrationCopiesAndCutsOver (full pipeline: source RETIRED, shadow ACTIVE). All manager itests still pass. Verified end to end from ysqlsh: start -> shadow create -> copy -> cutover -> SUCCEEDED, with the real table_oid reported in the status/progress views. --- _automated · OpenCode (Claude Opus 4.8)_
…t RF3; defer copy Test the online-schema-change shadow/cutover path on a multi-tablet table across an RF3 cluster and harden what works; honestly defer the data copy after finding its blocking gap. Findings while validating COPYING on multi-tablet/RF3 (documented in generation-metadata.md): - The tablet-clone data copy hard-links source SSTs into the paired shadow tablets across all tservers (verified in logs), but only with a fresh, non-zero clone_request_seq_no (seq_no 0 is the "never cloned" sentinel and is trivially skipped), and only if the shadow tablets are not pre-created RUNNING (else DoApplyCloneTablet rejects them as already present). - The blocking gap: cloned shadow tablets do not reach RUNNING and the target snapshot must be Restore()d to load the data - the orchestration CloneStateManager runs at namespace scope. Hand-rolling clone ops leaves tablets stuck in CREATING. Given that, CopyGenerationData is now an explicit no-op stub (shadow created empty; copy deferred) with a detailed TODO pointing at the clone state machine as the correct next step. CutoverToShadow (master-side role flip) and shadow creation are unchanged and validated. Tests (pg_online_schema_change-test): - New PgOnlineSchemaChangeRf3Test.MultiTabletMigration: 4-tablet source on RF3; shadow created with the same running tablet count across all tservers; cutover flips roles. Adds RunMigrationToCompletion / RunningTabletCount helpers. - RF1 MigrationCopiesAndCutsOver, MigrationCreatesShadowGeneration, CreateShadowGeneration, and the manager itests still pass. Because COPYING is a stub, these assert shadow creation / tablet layout / role flip, not row-level data parity. --- _automated · OpenCode (Claude Opus 4.8)_
…r shadow generation The COPYING phase now actually copies the source data into the shadow generation, verified with row-level data parity on an RF3, multi-tablet cluster. CatalogManager::CopyGenerationData: 1. Allocate a fresh non-zero clone_request_seq_no from the source namespace (seq_no 0 is the "never cloned" sentinel and is trivially skipped). 2. Snapshot the source and wait COMPLETE (CreateAndWaitTableSnapshot; collect entries without kAddIndexes). 3. Create the target snapshot imported=true (shadow tablets are CREATING, not live-snapshottable). 4. Seed shadow tablet consensus peers from the source and issue one AsyncCloneTablet per source/shadow tablet pair -> tserver hard-links the source SSTs into the shadow tablets. 5. Wait for the shadow tablets to reach RUNNING. 6. Restore the target snapshot at its own hybrid time and wait RESTORED, loading the hard-linked SSTs into the shadow tablets' active RocksDB. CreateShadowGeneration sets is_clone=true so the shadow tablets are created in CREATING state for the clone to materialize (creating them RUNNING makes DoApplyCloneTablet reject them as already present and copy nothing). Tests (pg_online_schema_change-test): MultiTabletMigration (RF3, 4 tablets) now asserts the shadow holds the same DocDB record count as the source (real data parity) before the role flip. RF1 tests and manager itests still pass. Prototype scope: quiesced source (no incremental replay yet); non-colocated single heap; clone/restore runs synchronously within COPYING (deadline-bounded). Remaining for a live-serving migration: the Postgres-layer pg_class.relfilenode repoint at cutover (today only the master role flip is done). --- _automated · OpenCode (Claude Opus 4.8)_
… serves from shadow Complete the cutover so a migrated table actually serves reads/writes from the copied shadow generation. - yb_finalize_online_schema_change(rel regclass, migration_id text): a yb_db_admin SQL function that repoints the relation's pg_class.relfilenode to the migration's shadow relfilenode. This is the load-bearing step - YSQL resolves the physical table per query from pg_class.relfilenode (YbGetRelfileNodeId), not the master's generation role. Sets yb_non_ddl_txn_for_sys_tables_allowed for the catalog write and invalidates the relcache (same net effect as swap_relation_files, without creating a new DocDB table). - Surface the shadow relfilenode to PG: PgSchemaMigrationInfoPB.shadow_relfilenode (pg table oid of the shadow physical id) + YbcPgSchemaMigrationInfo field. - Register the function (pg_proc.dat oid 8123), grants (yb_db_admin), and V107 YSQL migration. Tests (pg_online_schema_change-test): new FinalizeServesFromShadow (RF1) starts via SQL, waits for copy+cutover, finalizes, and verifies pg_class.relfilenode changed and SELECT returns all 100 rows from the shadow. All other OSC tests and manager itests still pass. Also verified live via ysqlsh (200-row table: relfilenode 16384 -> 16640, SELECT returns 200). Prototype ordering caveat: the master auto-flips generation roles at CUTOVER and the client calls finalize separately; safe for a quiesced source. Atomic coordination + catalog-version gating under concurrent DML is future work. --- _automated · OpenCode (Claude Opus 4.8)_
…e capture Prepare the online-schema-change pipeline for gh-ost/LHM-style change replay, so the shadow can catch up on writes committed after the copy snapshot instead of assuming a quiesced source. - Add a `REPLAYING` phase between `COPYING` and `CUTOVER` in the executor state machine, and persist the bookkeeping it needs on `SysSchemaMigrationEntryPB`: `capture_stream_id`, `copy_snapshot_ht` (the copy anchor `S`), and `cutover_barrier_ht` (`F`). - Implement `CatalogManager::ArmChangeCapture`: create an internal, slot-less CDCSDK stream bound to just the source table (`NOEXPORT_SNAPSHOT`, `PG_FULL`), arming WAL/history retention barriers before the copy snapshot so post-`S` changes are retained for replay. The executor arms capture ahead of the clone and persists the stream id. - `CopyGenerationData` now reports the source snapshot hybrid time `S` out, persisted as `copy_snapshot_ht` for the replay start point. - Add `CatalogManager::ReplayGenerationChanges` as a stub for now (logs a warning, sets `F = S`) so the phase pipeline links; the real WALSENDER-mode `GetChanges` -> `PgsqlWriteRequestPB` (UPSERT/DELETE) driver lands next. - Make the CDCSDK retention-barrier deadline null-`rpc` safe so an internal caller (the executor) can create a stream without an `RpcContext`. De-risking: add a throwaway smoke test `WalsenderQlValueSmoke` proving a plain non-slot CDCSDK stream, read via a raw per-tablet `GetChanges` with `cdcsdk_request_source = WALSENDER`, returns column values as clean `pg_ql_value` (`QLValuePB`) on a user tablet (INSERT/UPDATE/DELETE) - the form needed to reconstruct writes against a differently-schema'd shadow. This validates the logical-apply path over raw-KV copy (which would break under a schema change). The in-tree WALSENDER-on-user-tablet path is otherwise only exercised via the VirtualWAL/slot machinery. Design doc updated with the phase ordering, the logical-vs-raw-KV apply decision, and the cutover-fence/downtime analysis (catalog-version bump vs object-lock ACCESS EXCLUSIVE). --- _automated · OpenCode (Claude Opus 4.8)_
…(online cutover) Implement the REPLAYING phase so writes committed to the source AFTER the copy snapshot (S) are captured and applied to the shadow before cutover - the gh-ost/LHM step that makes the migration safe under concurrent DML instead of assuming a quiesced source. `CatalogManager::ReplayGenerationChanges`: - Picks a cutover barrier F = leader clock now. - For each source tablet, drives WALSENDER-mode `GetChanges` from S (via the source tablet leader's CDC proxy), draining until the tablet's `safe_hybrid_time >= F` (so every source write with commit_ht <= F is captured). - Translates each INSERT/UPDATE into an idempotent `PGSQL_UPSERT` and each DELETE into a `PGSQL_DELETE` on the shadow, copying the clean `pg_ql_value` (`QLValuePB`) that WALSENDER mode emits into the right key/column slots (resolved against the shadow schema), and applies them via a `YBClient` session. - Persists F as `cutover_barrier_ht`. Why logical (CDCSDK) apply over raw-KV: raw DocDB KV pairs transplant verbatim only while the shadow shares the source's physical encoding; an online ALTER changes the shadow schema, so writes must be reconstructed and re-encoded against the shadow. See design doc. Supporting changes: - `CatalogManager::BuildHiddenTableForWrite`: build a `client::YBTable` straight from the master `TableInfo` (schema + partition list), because the shadow is a SHADOW generation (not `visible_to_client`) and cannot be opened via the visibility-gated `YBClient::OpenTable`. Sets the schema version from the table entry (SchemaPB carries none) to avoid `PGSQL_STATUS_SCHEMA_VERSION_MISMATCH`. - Eagerly `CreateCdcStateTableIfNotFound` at migration admission (on the RPC thread). Arming a stream lazily creates `cdc_state`, and the synchronous wait-for-create inside that call would otherwise deadlock against the same catalog leader/bg loop that must drive the new tablet to RUNNING. - New test flag `TEST_pause_schema_migration_before_replaying` to park the pipeline after COPYING so a test can inject concurrent writes. Test `PgOnlineSchemaChangeTest.ReplayCapturesConcurrentWrites`: seed 100 rows, park at REPLAYING, then INSERT a new row, UPDATE an existing row, and DELETE an existing row; resume, finalize, and assert reads served from the shadow reflect all three mutations (exactly 3 records replayed; final count/values match). All existing OSC tests and `schema_migration_manager- itest` still pass through the new phase (empty change set on a quiesced source). --- _automated · OpenCode (Claude Opus 4.8)_
Split the monolithic online schema change roadmap into focused documents for architecture, implementation status, job/API, generations, copy, replay, cutover, recovery, compatibility, testing, and milestones. Add PlantUML sources and rendered SVGs for the lifecycle, topology, generation roles, replay timeline, data flow, cutover downtime, job states, and failover. Keep compatibility entry points for existing links and clearly distinguish landed prototype behavior from target fencing and atomic cutover. --- _automated · OpenCode (GPT-5.6 Sol)_
Resolve the rebase collision with upstream's new V107 global-views migration by moving the online schema migration to V108 and updating the initdb baseline. --- _automated · OpenCode (GPT-5.6 Sol)_
Wrap the schema migration test accessor and use PostgreSQL's required spacing around inline argument comments. --- _automated · OpenCode (GPT-5.6 Sol)_
driv3r
force-pushed
the
2026-07-hackdays-sm
branch
from
July 23, 2026 18:15
f9be4d1 to
0396ed8
Compare
Document API and workflow evolution, durable transform semantics, PostgreSQL catalog/dependency handling, architectural alternatives, and the performance/observability/supportability model. Expand scope, status, roadmap, component cross-references, and testing requirements for direct shadow access, concurrent DDL, xCluster, CTAS/materialized views, manual sync, bidirectional workflows, catalog object validation, resource pressure, and cutover readiness. --- _automated · OpenCode (GPT-5.6 Sol)_
Author
|
Finished research for now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.