⚠️ This project is under active development. See Current Status for progress.
A failure-aware streaming system with invariant-driven correctness,
validated through executable tests and later extended toward durable persistence, runtime semantic outcomes, and governance-oriented system reasoning.
This is not a CRUD or ETL portfolio demo.
It is a production-inspired streaming-system project focused on:
- accepted-history-first correctness
- semantic validation before persistence
- orthogonal idempotency / concurrency boundaries
- replay-safe projection runtime design
- exact-money hardening before durable persistence
- snapshot trust as derived-state replay efficiency, not source-of-truth replacement
The project currently has:
- a completed write-side transactional baseline
- Compass Layer 1 transition-truth validation
- a Stage 3 baseline projection runtime with reducer / worker separation
- a completed Stage 3.5A decimal / money hardening step before durable persistence
- a completed Stage 3.5B durable write-side baseline, including PostgreSQL-backed accepted history, durable idempotency, transactional write-side execution, two-phase concurrency admission, and validation placement strategy
- Stage 3.5C PR0 durable order-event vocabulary hardening, including uppercase
event_typevocabulary,proof_prev_statusdatabase constraint, and stream-position unique-constraint rename - a completed Stage 3.5C durable read-side baseline, including PostgreSQL-backed projection state, checkpoint progress, global-position projection worker orchestration, and durable replay / rebuild validation
- a completed Stage 3.5D snapshot trust / replay-efficiency baseline, including projection snapshot schema, store, replay validator, snapshot-assisted state resolver, and aggregate snapshot trust deferral
- a completed Stage 3.5E durable history and permission hardening baseline, including database role / privilege boundaries, accepted-history mutation hardening tests, derived-state mutation permission tests, and a minimal actor metadata boundary
- executable tests defending write-side, read-side, durable replay, snapshot trust, and durable permission-boundary semantics
Stage 3.5E is now complete at the minimal actor / permission boundary level.
The current implementation focus is:
- Stage 4 — runtime semantic governance
Stage 4 continues the roadmap toward runtime semantic validation, structured semantic outcomes, decision receipts, runtime decision policy, strategy selection, retry governance, and later action safety.
If you only have a short amount of time:
-
1 minute
- read Current Status
- scan the High-Level Architecture
-
3 minutes
-
5 minutes
- read Projection Boundary
- read Transactional Core
- scan
tests/and the Stage 3 projection worker / reducer path
If you want to understand how the repository thinks rather than only what it implements:
- read Learning and Design Methodology
- read Postmortems
-
Accepted-history-first design
Candidate events are not trusted merely because they can be written. Accepted history is protected by semantic validation and admission boundaries. -
Layered semantic defense
Compass is intentionally split into write-side transition-truth validation and later runtime state validation. -
Orthogonal idempotency and concurrency boundaries
Retry safety and stale-write protection are treated as different problems, not collapsed into one mechanism. -
Replay-safe projection baseline
The Stage 3 projection runtime uses reducer / worker separation and replay / rebuild through the same runtime path. -
Exact-money pre-persistence hardening
Stage 3.5A completed the migration from float-based money handling to Decimal-based semantics before durable persistence work expands. -
Durable write-side baseline
Stage 3.5B establishes PostgreSQL-backed accepted history, durable idempotency, transactional write-side execution, two-phase concurrency admission, and configurable validation placement. -
Durable read-side baseline
Stage 3.5C establishes PostgreSQL-backed projection state, checkpoint progress, global-position worker orchestration, and durable replay / rebuild validation against accepted history. -
Snapshot trust / replay-efficiency baseline
Stage 3.5D establishes projection snapshot schema, snapshot storage, snapshot-assisted replay validation, snapshot-assisted state resolution, and aggregate snapshot trust deferral. -
Completed minimal actor / permission boundary
Stage 3.5E completes the database role / privilege baseline, accepted-history mutation hardening, derived-state mutation permission tests, and minimal actor metadata boundary before Stage 4. -
Documentation as architecture memory
ADRs, boundary notes, postmortems, and philosophy notes are used to preserve why the system is shaped this way.
This project is a production-inspired streaming system designed to solve three fundamental problems:
-
Transactional Correctness
Ensure state transitions are logically valid. -
Runtime Semantic Integrity
Ensure derived runtime state remains faithful to accepted history. -
Failure Resilience under Adversarial Conditions
Maintain correctness even under failures.
One event stream, two semantic worlds
The same data, interpreted under different system semantics
- Transactional Pipeline → state transition
- Read-Side / Runtime Pipeline → derived runtime truth
flowchart TD
A[Commands / Requests] --> B[Transactional Pipeline]
B --> C[Compass Layer 1<br/>Transition Truth Validation]
C --> D{Concurrency / Admission Gate}
D -- Accepted --> E[(Event Log / Immutable History)]
D -- Rejected / Conflict --> R[Reject / Retry Flow]
E --> F[Projection Pipeline]
F --> H[Derived Runtime State]
H --> I[Compass Layer 2<br/>Runtime State Validation]
This architecture separates:
- write-side admission, where candidate events must pass semantic validation and concurrency-safe persistence
- accepted immutable history, where admitted events become the durable source of truth
- read-side derivation, where projection interprets accepted history into runtime state
- runtime verification, where Compass later validates whether projected state remains semantically correct
- Event-driven architecture
- Immutable event log as accepted history
- State as derived projection
- Invariant-driven validation through Compass
- Version-based admission and deterministic replay
- Clear separation between idempotency, concurrency control, and semantic validation
Invariant = State Compression + Contract
Compass is the semantic validation layer of the system.
It begins with event-level transition truth validation and evolves toward runtime state verification, structured semantic outcomes, and later governance-oriented decisions.
At a high level:
- Compass Layer 1 validates whether a candidate event truthfully follows accepted history.
- Compass Layer 2 validates whether projected runtime state remains semantically correct.
- Structured semantic outcomes are the later bridge from raw failure detection toward reusable governance meaning.
- Compass governance later decides how the system should respond to violations.
Compass does not replace concurrency control.
Instead:
- Compass decides whether a candidate event is semantically trustworthy.
- The admission gate decides whether that candidate can still become the next accepted fact.
- Idempotency decides whether the external request has already been processed.
This system is intended to be validated through failure injection, including:
- poison messages
- partial commit failures
- duplicate events
- out-of-order events
- race conditions
- network jitter
- backpressure
Chaos scenarios do not define correctness.
They test whether the correctness mechanisms inside src/ survive adversarial runtime conditions.
A system is not correct because it works.
A system is correct because it preserves truth under failure.
- Deterministic state recovery from accepted history
- Idempotent request handling with replay/conflict distinction
- Concurrency-safe event admission
- Candidate-event semantic validation before persistence
- Executable write-side invariants through tests
- Stage 3 baseline read-side projection runtime with reducer / worker separation
- Replay-safe projection state derivation with checkpoint-aware sequencing
- Decimal-based money semantics before durable persistence
- PostgreSQL-backed durable write-side persistence
- Two-phase concurrency admission through
prepare_stream(order_id)andappend_if_admitted(candidate_event, expected_current_version) - Configurable validation placement through
IN_TRANSACTIONandPRE_TRANSACTION - Durable order-event vocabulary hardening through uppercase
event_typevalues,proof_prev_statusCHECK constraints, and explicit stream-position constraint naming - Projection snapshot trust as derived compression, validated and resolved without making snapshots authoritative
- Clear separation between domain legality, transition truth, admission continuity, retry safety, validation placement, read-side derivation, and snapshot trust
- A CRUD system
- A simple ETL pipeline
- An AWS deployment demo
- A dashboard-first analytics project
A production-inspired streaming system focused on:
- correctness
- reliability
- failure modeling
- semantic validation
- replayable state reconstruction
- durable-boundary design before broader runtime complexity
streaming-system-compass/
├── src/ # Semantic core and execution logic
│ ├── core/ # Transactional domain core
│ ├── pipeline/ # Transactional / projection / analytical flows
│ ├── storage/ # Persistence abstractions
│ ├── compass/ # Semantic validation and governance
│ └── bootstrap/ # Composition roots / runtime assembly
├── chaos_engine/ # Failure injection and adversarial testing
├── experiments/ # Demo scripts and isolated experiments
├── docs/ # Philosophy, architecture notes, ADRs, domain specs, boundary notes, roadmaps, postmortems
├── tests/ # Unit, integration, replay, semantic-case, and adversarial baseline tests
├── README.md
└── .gitignore
From the repository root:
pip install -r requirements.txt
pytest -vRun a specific test directory:
pytest tests/integration -vSome PostgreSQL-backed integration tests require local PostgreSQL, TEST_DATABASE_URL, and migrations.
For the full local PostgreSQL setup and destructive integration-test guardrails, see:
The full documentation index starts at docs/README.md.
Key documentation areas:
- Design Philosophy — working methodology and mental models behind IBO, Core / Enabler separation, and Compass-style semantic alignment
- Architecture Notes — subsystem architecture and runtime boundaries
- Architecture Decision Records — major design decisions and trade-offs
- Domain Specifications — versioned business rules and domain semantics
- Boundary Notes — module ownership and responsibility boundaries
- Roadmaps — implementation sequencing and system evolution
- Postmortems — design lessons and boundary reflections
This project is guided by a small set of mental models:
- Input / Bridge / Output (IBO) for reasoning across function, pipeline, and system scales
- Core / Enabler separation for distinguishing business meaning from reliability mechanisms
- Map / Compass alignment for adapting static design to runtime disturbance
These notes are collected in Design Philosophy.
They are not implementation proof. They explain the reasoning model behind the architecture, while executable correctness belongs in src/ and tests/.
- High-Level Architecture
- Learning and Design Methodology
- Transactional Core
- Order Domain v1 Rules
- Stateless Registry and Concurrency Strategy Boundary
- Concurrency Control, Idempotency, and Retry Safety
- Intent-Aware Validation Dispatch for Compass Runtime
- Why Compass Split into Two Layers
- Compass Layers
- Projection Pipeline
- Implementation Roadmap
- Compass Runtime Roadmap
- Boundary Notes
- Postmortems
This order starts from the top-level system shape, then moves into working methodology, write-side semantics, domain rules, transactional safety decisions, Compass validation architecture, projection evolution, and implementation sequencing.
For the mental models behind the architecture, see Design Philosophy, especially the notes on learning/design methodology, IBO, and Core / Enabler separation.
The implementation begins from the transactional semantic core under src/core/order/.
This means the project does not start from chaos injection, dashboards, analytics, or cloud deployment.
Instead, it starts by defining and implementing:
- domain event semantics
- aggregate rules
- state transitions
- proof / provenance structure
- idempotency boundary
- concurrency-safe admission boundary
- core transactional invariants
Everything else grows around this core:
storage/persists accepted history, idempotency memory, and protects version continuitypipeline/executes transactional and projection flows, including PostgreSQL-backed write-side orchestrationcompass/validates semantic correctnessbootstrap/assembles concrete runtime wiringchaos_engine/stress-tests whether mechanisms insidesrc/survive adversarial conditions
- transactional domain core
- event generation and replay
- idempotent request handling
- concurrency-safe admission / conditional persistence
- write-side consistency baseline
- proof-carrying event structure
- event-level Compass validation
- transition truth checking before persistence
- validation dispatch and basic
ALLOW/BLOCKpolicy
- pure reducer
- checkpoint-aware projection worker
- projection state store
- checkpoint / offset handling
- replay and rebuild flow
- Decimal hardening before durable persistence
- Stage 3.5B durable write-side baseline completed
- Stage 3.5C durable read-side baseline completed
- PostgreSQL-backed accepted history and idempotency memory
- transactional event append + idempotency record persistence
- two-phase concurrency admission and validation placement strategy
- Stage 3.5C durable read-side schema, stores, worker orchestration, and replay validation completed
- durable projection state and checkpoint state
- global-position projection worker orchestration
- persistence-backed replay / rebuild validation
- Stage 3.5D snapshot trust, replay-efficiency, and persistence optimization completed
- pre-Stage 3.5E documentation alignment completed
- Stage 3.5E minimal actor / permission boundary completed
- exact money durability is already established at the baseline level
- durable accepted-history and permission hardening are now established as the pre-Stage 4 baseline
- projected state invariants
- replay vs incremental consistency checks
- Layer 2 minimal validator
- structured semantic outcomes
- future Layer 1 / Layer 2 outcome-family alignment
- reviewer-friendly demo packaging
- documentation alignment
- clear implementation vs future-work boundary
- portfolio / open-source-ready milestone
- advanced governance policy actions
- warning / quarantine / audit behavior
- evidence logging
- semantic alerts
- adversarial failure validation through chaos scenarios
This repository is being built incrementally toward the full system design described above.
Current baseline completed:
- transactional semantic core under
src/core/order/ - minimal
INIT -> CREATED -> PAIDwrite-side model - accepted-history event store and replay baseline
- request-level idempotency with replay/conflict distinction
- optimistic admission gate for append-time continuity protection
- optimistic concurrency collision coverage for stale-write rejection
- Compass Layer 1 transition-truth validation
- runtime assembly through
src/bootstrap/ - Stage 3 baseline projection runtime:
- pure reducer
- checkpoint-aware worker
- in-memory projection state store
- in-memory checkpoint store
- replay / rebuild baseline
- Stage 3.5A exact-money hardening before durable persistence:
- Decimal-based money semantics
- aligned fixtures / unit / integration / semantic / adversarial / demo paths
- formal projection reducer path as the only replay-reduction truth path
- Stage 3.5B durable write-side baseline:
- PostgreSQL-backed accepted history through
PostgresEventStore - PostgreSQL-backed idempotency memory through
PostgresIdempotencyStore - transactional event append and idempotency record persistence
- two-phase concurrency admission through
prepare_stream(order_id)andappend_if_admitted(candidate_event, expected_current_version) - validation placement strategy for
IN_TRANSACTIONandPRE_TRANSACTION
- PostgreSQL-backed accepted history through
- Stage 3.5C durable read-side baseline:
- durable order-event vocabulary hardening
- durable read-side schema for
projection_statesandprojection_checkpoints - PostgreSQL-backed
PostgresProjectionStore - PostgreSQL-backed
PostgresCheckpointStore order_events.global_position- PostgreSQL-backed projection worker orchestration
- durable replay / rebuild validation against accepted history
- PostgreSQL schema, storage, worker, and replay-validation tests
- Stage 3.5D snapshot trust / replay-efficiency baseline:
- snapshot trust contract boundary
- durable projection snapshot schema
- PostgreSQL-backed
PostgresProjectionSnapshotStore - projection snapshot-assisted replay validator
- projection snapshot-assisted state resolver
- aggregate snapshot trust boundary / deferral decision
- write-side aggregate snapshot schema/store and snapshot-assisted write-side rehydration explicitly deferred
Current boundary of completion:
- write-side transactional baseline is established
- durable write-side persistence is established
- read-side projection baseline exists in deterministic in-memory form
- exact-money semantics are stabilized before deeper durable persistence work
- durable accepted-history vocabulary has been hardened before read-side persistence depends on stored events
- durable read-side projection state, checkpoint progress, global-position worker orchestration, and replay / rebuild validation are established through Stage 3.5C
- Stage 3.5D snapshot trust, persistence optimization, and replay efficiency is complete at the read-side projection snapshot baseline level
- write-side aggregate snapshot schema/store and snapshot-assisted write-side rehydration are intentionally deferred
- Stage 3.5E durable history and permission hardening is complete
- state-level Compass Layer 2 validation is not yet implemented
Current implementation focus:
- Stage 4 runtime semantic governance
Stage 3.5E closeout is complete at the durable history / permission / actor-boundary level.
Next implementation milestones:
- Stage 4 runtime semantic validation, semantic outcome structuring, decision receipts, runtime decision policy, strategy selection, and retry governance
- Stage 5 dual-dimension governance demo / action safety
This repository began with a documentation-first development approach.
The architecture, ADRs, domain rules, and boundary notes were written before the main transactional implementation to make ownership, invariants, and failure boundaries explicit.
That documentation-first phase has now been translated into an executable baseline across:
src/core/order/src/storage/src/pipeline/transactional/src/pipeline/projection/src/compass/transition/src/bootstrap/tests/
The repository remains intentionally conservative:
- documentation defines semantic intent and ownership boundaries
src/implements runtime logictests/make selected invariants and failure paths executable- the current Stage 3 baseline remains intentionally minimal and in-memory
- Stage 3.5A has hardened exact-money semantics before persistence expands
- later phases will extend this baseline toward durable persistence, runtime semantic outcomes, and adversarial hardening
This repository is shared as a personal design research project and professional portfolio.
Unless otherwise noted:
- Source code is not currently licensed for reuse, redistribution, or modification.
- Documentation, notes, diagrams, and written materials are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
You may share and adapt the written materials with proper attribution.
Please attribute written materials as:
Compass Framework / Streaming System Compass documentation by Yen-Hua Chen.
Licensed under CC BY 4.0.
Original source: https://github.com/hikaru-212/streaming-system-compass
For usage, redistribution, attribution, and permission details, see NOTICE.md.
For the documentation content license text, see LICENSE-CONTENT.md.
This project focuses on system correctness under failure, not just successful execution under ideal conditions.
The main logic of correctness lives in src/.
chaos_engine/ exists to test whether those correctness mechanisms can survive real failure conditions.
The documentation follows one main principle:
Explain the boundary before explaining the implementation.