Conversation
…odel Add the knowledge-layer scoring foundation: core types with msgpack serialization, DDL parser for all 14 statement types, SchemaManager CRUD operations, compiled binding table, Badger key prefixes 0x11-0x17, schema persistence round-trip, and feature flag gating via DecayEnabled. Includes 141 tests across pkg/knowledgepolicy, pkg/storage, and pkg/cypher. Five parser/validation bugs discovered and fixed during test development (see implementation plan deviations).
Add the pure-computation scoring layer: binding builder compiles decay profiles into a BindingTable with pre-computed ThresholdAgeNanos, resolver handles multi-label subset resolution with Order-based tie-breaking, and scorer implements all four decay functions (exponential, linear, step, none) with the normative promotion formula. 62 tests + 4 benchmarks. Disabled fast path: 13ns/0 allocs.
Add P-local sharded access accumulator with sync.Pool P-affinity for zero-allocation, zero-contention hot-path accumulation (42ns/0alloc single-entity, 87ns/0alloc parallel, 1ns disabled). Add access flusher goroutine that drains P-local shards and merges accumulated deltas with persisted Badger entries via AccessMetaStore interface. Add Badger AccessMeta CRUD layer (0x11 prefix) with msgpack encoding for Get/Put/Delete/Scan operations. Inline Kalman filter from pkg/filter/kalman.go for zero-allocation mutation processing (16ns/0alloc). Includes velocity projection, error boosting, auto-R variance tracking with sliding window, and manual-R fixed-noise modes. Add anti-sycophancy test suite validating hallucinated spike dampening, sustained sycophancy smoothing, and genuine trend tracking through the Kalman filter.
…g-before-visibility
Wire the knowledge-layer scorer into all read paths so nodes, edges, and
properties are evaluated for decay suppression before becoming visible to
queries. This is the first phase where decay actually affects query results.
Key changes:
- ShouldSuppressNode/Edge pure visibility predicates with legacy field fallback
- filterNodeByDecay/filterEdgeByDecay on BadgerEngine read paths (GetNodesByLabel,
AllNodes, 6 MVCC methods)
- BindingTable synchronous rebuild on every DDL mutation and schema load
- reveal() Cypher function and per-query revealAll bypass via atomic bool
- Property-level decay filtering in nodeToMap result projection
- AccessFlusher property suppression with embed invalidation callback
- Search candidate decay filtering before RRF fusion
- DB lifecycle wiring: accumulator, flusher, and BadgerEngine decay enablement
42 new tests across knowledgepolicy, storage, cypher, and search packages.
…e, decay, policy Expose the scoring system to Cypher queries via three inline-dispatch functions: decayScore(entity) returns the decay score as a float, decay(entity) returns the full scoring resolution as a map, and policy(entity) returns the AccessMeta entry. All three degrade gracefully when decay is disabled, returning neutral values with a once-per-query config mismatch warning.
…eindex infrastructure Add three-layer suppression system: VisibilitySuppressed fast-path bit on Node/Edge structs, index tombstones (prefix 0x17) for key-only scans, and a background DeindexCleanupJob that drains work items and writes tombstones. IndexEntryCatalog (prefix 0x12) tracks exact secondary-index keys per entity so the cleanup job never needs full index scans.
…ay model
Replace the domain-specific Memory and Edge types with the generic labeled
property graph API (Node/Edge). The knowledge-layer policy system now handles
all scoring and visibility through declarative profiles, making hardcoded
tier/decay semantics obsolete.
- Delete pkg/decay/ entirely (hardcoded three-tier decay manager)
- Delete Memory struct, Edge struct, and 6 legacy methods (Store, Recall,
Remember, Link, Neighbors, Forget) from pkg/nornicdb/
- Delete db_memory_mapping.go and BootstrapCanonicalSchema
- Remove MemoryTier enum from GraphQL schema, models, and generated code
- Remove tier-specific fields from storage types and replication codec
- Add storage migration framework (v0→v1) for legacy node format
- Update MCP server: accept caller-specified labels array, pass edge
metadata through as properties, use configurable default node label
(NORNICDB_DEFAULT_NODE_LABEL env var, defaults to "Memory")
- Make flush interval and buffer size configurable in knowledgepolicy
- Rename ArchiveThreshold → VisibilityThreshold across all layers
- Rewrite all tests to use generic CreateNode/GetNode/CreateEdge APIs
…umentation, and suppression anchors
Phase 8 completes the knowledge-layer scoring system with admin endpoints,
browser UI, documentation suite, and the Ebbinghaus-Roynard bootstrap.
Adds a suppression anchor semantic: property-level NO DECAY on a decaying
node/edge blocks entity suppression. The HasNoDecayProperty flag is
pre-computed at binding compilation time for zero-cost scorer hot-path
checks. Redundant NO DECAY on already-permanent entities is removed from
the bootstrap.
Admin endpoints: /admin/knowledge-policies/{profiles,policies,resolve,deindex/status}
Browser UI: Knowledge Policies admin page + node detail panel decay section
Documentation: 5 new user guides including Ebbinghaus-Roynard bootstrap
Tests: 20 bootstrap, 14 property visibility, 8 Kalman multi-agent, 15 e2
…r concurrency Fix several correctness issues across knowledge policy, Cypher reveal handling, storage migrations, config naming, and embed worker lifecycle. Changes: - use namespaced memory env vars for access flush buffer and visibility threshold - fix decay diagnostics to report resolved function and effective floor - guard effective decay rate when half-life is unset or zero - correct fact_nodecay integration test profile key - use selected database decay state in knowledge policy server handlers - make reveal mode query-scoped to avoid shared-engine leakage across requests - fall back node version timestamps to createdAt when UpdatedAt is unset - propagate schema version read errors and reject invalid stored lengths - serialize embed worker Reset/Close lifecycle to prevent WaitGroup reuse panic - add focused regressions for reveal scope, decay dispatch, schema version errors, config env loading, node timestamp fallback, and embed worker concurrency
Remove the brittle decay() dispatcher guard that rejected any argument containing the keyword "score", which could block valid identifiers like scoreNode. Keep decayScore() on its explicit dispatch path and add a regression test proving decay(scoreNode) is handled correctly.
Update the installation guide to use the supported knowledge-policy DDL syntax for decay profile bundles and bindings instead of the old retention-binding examples. Make nornicdb.decay.info() report the actual decay-enabled state from the underlying engine when available, and update its configuredVia text to reference the supported CREATE DECAY PROFILE forms. Add regression checks for the procedure output.
…pression invalidation Implement ON ACCESS mutation execution in the access flusher, including expression evaluation, edge property scoring, and metadata lookup for both nodes and edges. Wire suppression rechecks through DB startup so entities are deindexed and Cypher caches are invalidated only for affected labels/types when suppression transitions occur. Tighten storage read paths to honor node, edge, and property suppression consistently, add end-to-end and regression coverage for access-driven decay and noisy corroboration promotion flows, and fix multidb storage-size tracking to account for persisted post-write entity shapes. Refs #100 Refs PR #117
…y defaults Fix schema lock leak in schema_knowledgepolicy.go: replace deferred unlock with finishKnowledgePolicyMutationLocked(), ensure early-return paths unlock, persist after unlock, and invoke knowledgePolicyChanged hook. Add SetKnowledgePolicyChangedHook to schema.go and wire it in badger_schema.go to trigger decay reconciliation when schema changes. Implement namespace-wide decay reconciliation: badger_decay_reconcile.go: ReconcileDecaySuppression and ReconcileDecaySuppressionWithChanges. Hook into BadgerEngine to enqueue de-indexing / produce suppression-change tokens. Move scoring and promotion logic forward: binding_builder.go / compiled_binding.go: compile promotion WHEN clauses into CompiledPromotionRule and attach to CompiledBinding. scorer.go: add ScoreNodeWithProperties / ScoreEdgeWithProperties, pass entityProps into scoring, and implement selectPromotionProfile() to run compiled WHEN predicates. binding_builder.go: add effectiveBundle* helpers and stricter handling of bundles with DecayEnabled. Fix ON ACCESS expression runtime: on_access_runtime.go: include entityProps in eval context, make missing numeric comparisons evaluate false for ordered ops, and update property lookup to consult both access meta and entity props. Make decay semantics explicit in DDL and tests: knowledgepolicy_ddl.go: parse decay bundles with DecayEnabled: true by default when appropriate. Update many tests to set DecayEnabled explicitly (badger_mvcc_decay_test.go, pkg/cypher/*_test.go, pkg/knowledgepolicy/*_test.go, integration_test.go). Add bootstrapping & wiring for default knowledge-policy: knowledgepolicy_bootstrap.go: bootstrap DDL when opening a decay-enabled DB with an empty namespace. Wire bootstrap and knowledge-policy changed behavior into db.go. Add E2E tests for bootstrap and ALTER-driven unsuppression (knowledgepolicy_bootstrap_e2e_test.go, badger_knowledgepolicy_alter_e2e_test.go). Update storage filtering to use the new scoring API: badger_decay_filter.go: replace older filter calls with Score*WithProperties and use returned SuppressionEligible. Add/adjust unit & e2e tests to prevent regressions: schema_knowledgepolicy_test.go: add TestKnowledgePolicyMutationErrorsDoNotLeakSchemaLock. Several other test updates and new e2e tests to validate behavior. Test status: focused schema tests and the full storage package pass locally after these changes.
- split the schema-version and edge-between-ready MVCC meta keys - recover legacy one-byte ready markers during startup migrations - keep shared edge-between head keys out of per-edge index catalogs - honor tombstones and decay filtering in edge-between lookup paths
Add knowledge-policy DDL routing and built-in procedures so the UI can manage and inspect decay and promotion configuration through Cypher instead of the admin HTTP API. Changes: - route CREATE/ALTER/DROP/SHOW knowledge-policy commands through the dedicated DDL execution path - add CALL nornicdb.knowledgepolicy.info/profiles/policies/resolve/ deindexStatus procedures - expose row-shaped SHOW results for decay profiles and promotion policies - migrate UI knowledge-policy client methods from admin endpoints to Cypher calls while preserving existing response shapes - add unit and e2e coverage for the new procedure and DDL surfaces
…rty, fix docs import Agent-Logs-Url: https://github.com/orneryd/NornicDB/sessions/87acf3af-aa82-41f4-9ae8-fe47f62c6a8f Co-authored-by: orneryd <1736223+orneryd@users.noreply.github.com>
Owner
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release Summary
This release completes the new knowledge-layer scoring system and moves it onto a single Cypher-first surface and provides a proper otel service endpoint configuration for monitoring.
this resolves issue #100, #117 , and #126
New features
CREATE/ALTER/DROP/SHOW DECAY PROFILECREATE/ALTER/DROP/SHOW PROMOTION PROFILECREATE/ALTER/DROP/SHOW PROMOTION POLICYdecayScore(entity)decay(entity)policy(entity)reveal(entity)ON ACCESSbehavior with targeted promotion logic andWITH KALMANsupport for smoothing behavioral signals.CALL nornicdb.knowledgepolicy.info()CALL nornicdb.knowledgepolicy.profiles()CALL nornicdb.knowledgepolicy.policies()CALL nornicdb.knowledgepolicy.resolve(...)CALL nornicdb.knowledgepolicy.deindexStatus()NORNICDB_MEMORY_DECAY_ENABLED=trueis correctly propagated into the live engine.Cleanup and compatibility changes
/admin/knowledge-policies/*HTTP routes so Cypher is now the only supported knowledge-policy control and diagnostics surface.Documentation
Operator usage examples
TODO: