rebuild BM25 from nodes and harden index invariants

[xav-db]

Greptile Summary

This PR hardens the BM25 full-text search subsystem by introducing a reverse index (doc_id → list of (term, tf)) alongside the existing inverted index, enabling correct and efficient delete/update operations without re-scanning the inverted index. It also adds schema versioning so that on startup, if the stored index was built without the reverse index, the migration layer automatically clears and rebuilds the entire BM25 index from nodes_db. The add_n, update, and upsert traversal operators are updated to maintain BM25 invariants as nodes are created or changed.

Key changes:

• bm25.rs: Adds ReversePostingEntry, DocPresence/DocState enums, reverse_index_db (DUP_SORT), schema version read/write, and refactored insert_doc / delete_doc / update_doc that enforce strict consistency invariants (returns errors on index corruption).
• storage_migration.rs: migrate_bm25 clears and rebuilds the BM25 index in 1 024-node batches when the stored schema version doesn't match BM25_SCHEMA_VERSION = 2.
• add_n.rs: bm25.insert_doc is called after writing the node — but without a guard that checks whether the nodes_db write succeeded, allowing BM25 errors to mask prior errors.
• update.rs / upsert.rs: BM25 is correctly updated before (update) or after (upsert new node) the nodes_db write, with proper error propagation.
• Comprehensive new tests cover the reverse index, schema migration, update-to-searchable, and invariant enforcement.

Important Files Changed

Filename	Overview
helix-db/src/helix_engine/bm25/bm25.rs	Core BM25 rewrite adding a reverse index (doc_id → terms) and schema versioning. Logic is sound overall; minor duplication between reverse_entries and reverse_entries_rw.
helix-db/src/helix_engine/traversal_core/ops/source/add_n.rs	BM25 insert_doc is called unconditionally after nodes_db write, even on prior failure — can mask the original error if BM25 also fails.
helix-db/src/helix_engine/storage_core/storage_migration.rs	Adds migrate_bm25 that clears and rebuilds the BM25 index from nodes_db. Holding read_txn alive across batch write transactions can cause LMDB freelist growth; nodes without properties are silently skipped (consistent with add_n but deserves a comment).
helix-db/src/helix_engine/traversal_core/ops/util/update.rs	Correctly calls bm25.update_doc before nodes_db write; BM25 error short-circuits the node save, and both are in the same transaction so rollback is safe.
helix-db/src/helix_engine/traversal_core/ops/util/upsert.rs	BM25 insert_doc (new node) and update_doc (existing node) are correctly integrated into upsert_n; error propagation follows the existing ? pattern.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant migrate_bm25
    participant BM25Index
    participant nodes_db
    participant add_n / update / upsert

    Note over Caller, BM25Index: Startup Migration Path
    Caller->>migrate_bm25: migrate(storage)
    migrate_bm25->>BM25Index: schema_version(read_txn)
    alt version matches BM25_SCHEMA_VERSION
        BM25Index-->>migrate_bm25: Some(2) — skip
    else outdated or missing
        migrate_bm25->>BM25Index: clear_all(write_txn)
        migrate_bm25->>nodes_db: iter(read_txn) — batch by 1024
        loop Each batch
            migrate_bm25->>BM25Index: insert_doc per node with properties (write_txn)
        end
        migrate_bm25->>BM25Index: write_schema_version(2, write_txn)
    end

    Note over Caller, BM25Index: Normal Write Path
    Caller->>add_n / update / upsert: write op (RwTxn)
    add_n / update / upsert->>nodes_db: put node
    add_n / update / upsert->>BM25Index: insert_doc / update_doc / delete_doc
    BM25Index->>BM25Index: update inverted_index_db (term→postings)
    BM25Index->>BM25Index: update reverse_index_db (doc_id→terms)
    BM25Index->>BM25Index: update doc_lengths_db
    BM25Index->>BM25Index: update term_frequencies_db
    BM25Index->>BM25Index: update metadata (total_docs, avgdl)
    add_n / update / upsert-->>Caller: Result<TraversalValue>

Loading

Comments Outside Diff (1)

1. helix-db/src/helix_engine/traversal_core/ops/source/add_n.rs, line 141-148 (link)

   BM25 insert runs unconditionally, can mask prior errors

   The BM25 insert_doc call (lines 141-148) runs even when a previous operation failed (secondary-index insertion or nodes_db.put_with_flags). If bm25.insert_doc then also fails, it silently overwrites the original error stored in result, making the caller receive a BM25 error instead of the real nodes_db or secondary-index error. Adding an early-exit guard fixes both the error masking and the wasted BM25 work:

_{Last reviewed commit: 8d28406}