feat: worktree cache reuse + root config, scan-filter tuning, code embeddings off by default#37
Closed
naamanhirschfeld-armis wants to merge 460 commits into
Closed
Conversation
The Codex plugin manifest had display metadata but no MCP server, so installing
it exposed zero tools, and Goldziher/basemind shipped only a Claude-format
marketplace, so adding it in Codex listed nothing.
- .codex-plugin/.mcp.json (new): declare the basemind server via
${PLUGIN_ROOT}/scripts/mcp-launch.sh serve, mirroring the other harnesses.
- .codex-plugin/plugin.json: point mcpServers at it.
- .agents/plugins/marketplace.json (new): Codex-native marketplace listing with
the required source{}/policy/category shape (repo-rooted github source).
- sync-to-codex-plugin.sh: ship mcp-launch.sh (was excluded with all of scripts/).
- .gitignore: track the two new files; README: correct the Codex install rows.
The new Codex marketplace listing carries plugins[0].version; add it to release-bump.sh's bump + validation pass so it doesn't go stale on the next release (release-versioning lockstep rule).
When an agent re-reads a file it already saw, emitting only the changed lines saves most of the tokens. Add a stateless LCS line-diff primitive (delta.rs) and a `basemind delta --old <path>` subcommand (new from stdin): it emits a `+A/-R` header plus only the changed lines, with a 50KB / 2000-line bail to full content on pathological input and a "# unchanged" short-circuit. Pure std, no diff crate. The stateful read-cache hook that calls this is a later slice.
~45 tool schemas cost the agent tokens every session. Add an opt-in lean mode that exposes just 3 wrapper tools — list_tools, get_tool_schema(tool_name), invoke_tool(tool_name, tool_input) — deferring full schemas until requested. rmcp 1.7's #[tool_handler] only generates list_tools/call_tool/get_tool when the impl doesn't define them, so these are hand-written to branch on the env flag: unset/0/off (default) runs the macro-equivalent full surface unchanged; on routes through 3 wrappers, with invoke_tool dispatching to the real tool via the SAME ToolRouter (no tool logic duplicated). Default surface is provably identical (smoke test asserts flag-unset lists all tools + invoke_tool round-trips byte-identical to a direct call).
inbox_read/room_history errored `decode error: missing field 'seq'` when a stale
pre-W7 comms daemon (long-running singleton) served bare MessageMeta to a W7
client that expects SeqMeta{seq, meta}. W7 changed the History/Inbox element type
without bumping PROTO_VER, so the Hello/Welcome guard let the skew through.
Make SeqMeta decode-tolerant: `#[serde(flatten)]` meta + `#[serde(default)]` seq,
so a legacy bare-MessageMeta payload decodes with seq=0 (a safe sentinel — nothing
divides/indexes on it). Additive serde shim per schema-and-blob-compat; no proto
bump or store wipe needed.
`compress` returns a code file's outline (signatures, no bodies). `expand` is the companion for the context-offloading pattern: given path + symbol name (+ optional kind) it returns the full source body sliced from that symbol's L1 byte range, so an agent can compress a file to its outline and pull back only the implementations it needs. Exact-name match; multiple matches return an invalid_params error listing the candidates (forces disambiguation, never the wrong overload); body capped at 128 KiB with `truncated`. Smoke + harden coverage; README row.
When an agent re-reads a file it already read this session, serve a compact `basemind delta` line-diff instead of the full content. PostToolUse/Read hook, opt-in via BASEMIND_DELTA_READS=1 (mirrors the output compressor), with a per-session content cache keyed by path hash under ~/.cache/basemind/read-cache/. Fail-open at every step — first read, missing binary/jq, a too-large/unchanged diff, or a diff that isn't meaningfully smaller all leave the full read intact. Activates once a release binary ships the `delta` subcommand.
…yspaces Foundation for code-grounded memory auto-curation (W10) and git-mined skill proposals (W11). Extends MemoryRecord with #[serde(default)] provenance, verified, last_verified, and importance fields — old field-absent msgpack blobs decode unchanged (round-trip tests prove it), so no INDEX_SCHEMA_VER / RELEASE_MINOR bump. Adds two lazily-created Fjall keyspaces (memory_archive, proposals) plus proposal key encoders; new keyspaces are append-only and do not trip the wipe-on-mismatch flow. importance is git-derived only — there is structurally no LLM rating field, so the importance-inflation failure mode is impossible by construction.
Verifies stored memories' code references against the live index and is the headline of the governance feature: no other memory system checks memory against code. File provenance → Stale when the file is gone; symbol provenance → Stale when the symbol is missing or its structural hash drifts (HashMode::Structural, formatter/comment-stable so reformatting never false-flags); commands are advisory only. Two triggers: an on-demand `memory_audit` MCP tool (single-key or scope range scan, dry_run preview, include_archived) and a best-effort, fail-open pass injected into scan_and_refresh after every rescan. Stale records decay importance and are auto-archived to the memory_archive keyspace after 90 days (recoverable, never deleted). The rescan pass is scope-prefix bounded so it never touches another repo's memories, and reuses the on-demand decay/archive logic. Structural-hash recompute path is cold in v1 (only symbol provenance triggers it); its perf hardening is tracked for when symbol capture ships.
Mines co-change clusters from recent git history (association-rule:
support + confidence over file pairs, bulk-commit guard, depth-1 transitive
clustering) into candidate skills, surfaced for one-tap accept/reject — never
auto-written. Importance is derived from git counts only (no LLM rating, so
inflation is impossible). Candidate ids are content-addressed (blake3 of the
sorted file-set) so re-mining is idempotent; rejected ids get a tombstone that
suppresses re-proposal.
Tools: proposals_mine, proposals_list (paginated), proposal_accept (promotes a
candidate to a searchable skill-tagged memory with file provenance, stamped
verified via the W10 audit engine, embedded into LanceDB), proposal_reject.
CLI parity via `basemind governance {mine,proposals,accept,reject}`; the five
identical cli::run tool arms in main.rs collapse into one dispatch closure to
stay under the line cap.
The smoke test proves the end-to-end code-grounded-staleness wedge
deterministically: accept a candidate -> its memory gains file provenance ->
delete a referenced file -> rescan -> memory_audit flips it to "stale" (then
the file is restored so later assertions see a pristine fixture). This is the
W10b-gap test that memory_put could not write.
rc.27's dependency tree transitively enables serde_json's `preserve_order` feature, flipping `Value` maps from sorted (BTreeMap) to insertion order. The native TOON encoder depended on that implicit sort for deterministic, byte-stable output, so make `encode_envelope` + `table_parts` sort keys/columns explicitly — keeping TOON output identical and independent of however the feature graph resolves. Full-feature build, lib tests, config-schema snapshot, and mcp_smoke all green on rc.27.
Close the W10 memory-curation test gaps with 16 unit tests plus a testable-core split of the rescan pass: - audit_one_record — every verdict branch: file deletion, symbol file-gone / name-absent, structural-hash drift (Stale) vs formatting-only edit (still Verified, the must-not-flag wedge), command-advisory (never Stale), empty provenance (Unverified). - evaluate_one — STALE_DECAY halving, the 90-day archive threshold, and dry_run / from_archive no-mutation guarantees, with exact-value asserts. - audit_scope_persist — extracted from audit_scope_on_rescan so the background pass is testable without a ServerState. Seeds crafted records straight into Fjall and reads the persisted row back, proving the pass durably flips Stale + decays, archives >90-day-stale rows out of the live keyspace, and never touches a foreign scope's records (the prior MCP-only attempt could not prove any of this — a dry-run audit recomputes the verdict, so it passed vacuously). Tests relocated to src/mcp/helpers_governance/tests.rs to keep the parent under the 1000-line cap; AuditCtx widened to pub(super) for the extracted core.
Cover the governance CLI surface end-to-end under --features memory: mine a co-change candidate, list it, accept it (verifying the `skill/cochange-` key + `skill`/`cochange` tags round-trip through `memory get`), reject a re-mined candidate, and confirm the tombstone suppresses it on the next mine. Plus a default-features test asserting a memory-gated subcommand fails gracefully (non-zero, no panic) when the feature is off. Known issue (documented in-test, tracked separately): `governance accept` writes the skill correctly but the process then exits 101 — kreuzberg's SharedEmbedder inner runtime is dropped inside the single-shot CLI runtime's blocking context (the serve path is unaffected). The test asserts on the persisted data and on a follow-up `memory get`, not on the accept exit code.
New tests/governance_smoke.rs over the serve MCP path: - support + confidence thresholds gate emission exactly at the boundary (cochange=5 rejected at min_support=6 / emitted at 5; confidence≈0.454 rejected at 0.5 / emitted at 0.4). - skipped_bulk counts an over-cap commit and its pairs never inflate co-change (a bulk-only pair is never mined). - proposal ids are deterministic across repeated mines (content-addressed). - proposals_list paginates (limit=1 -> truncated + next_cursor -> disjoint page 2) and filters by kind (memory -> empty in v1). - reject is idempotent and the tombstone suppresses the cluster on re-mine. Replaces the earlier vacuous rescan-audit test (a dry-run audit recomputes the verdict, so it could not prove the background pass; that proof now lives in the helpers_governance unit tests).
Promote the django proposals_mine sweep to a >= 1 canary. At the harness default thresholds (window=100, min_support=5, min_confidence=0.6) django's co-change history yields several candidates (measured ~4 on a depth-250 clone). The canary only enforces when mining actually ran — the value is absent on default-feature runs (proposals_mine returns not_enabled), so it is a no-op unless the harness is built with --features memory.
Kimi Code: kimi.plugin.json at the repo root (so the plugin root resolves to the repo, reusing skills/ and scripts/mcp-launch.sh) wiring the MCP server, skills, and a sessionStart bootstrap. Kimi manifests support mcpServers but not hooks, so the comms auto-injection is intentionally omitted there. pi: .pi/extensions/basemind.ts registers the skills dir and injects a session-start bootstrap; pi has no native MCP, so basemind is driven through its CLI there. Declared via the root package.json pi manifest. Wire kimi.plugin.json into release-bump.sh version lock-step (bump + validation).
Merge Quickstart + Installation into one section (plugin -> MCP -> CLI) with per-harness install docs for every supported harness, researched from each tool's official docs; add Kimi Code, Antigravity, and pi. Audit fixes: add Governance to the Capabilities pillars + a Governance CLI table; document the lang / hook / compress-output / delta CLI subcommands; merge Context-economy + Token-reduction into one Token-economy section; move the demo GIFs to a Demos section; drop the invalid 'watch --no-serve' flag; reconcile the compress roadmap with the shipped soft target_tokens hint. Bump the stale '8 coding-agent harnesses' string to '10+' across all shipped manifests.
… 101 The CLI runs each command inside an outer multi-thread `block_on`. `governance accept` (and `memory put`) materialise the cached `LanceStore`, which owns a current-thread tokio runtime. That store is dropped at the end of the async block — i.e. inside an async context — and the default `Runtime` drop blocks on its blocking pool, panicking with "Cannot drop a runtime in a context where blocking is not allowed" and exiting 101 even though the data write already succeeded. Hold the runtime in an `Option` and tear it down via `Runtime::shutdown_background` in `LanceStoreInner::drop`, tokio's sanctioned escape hatch for dropping a runtime from within another runtime. Harden the cli_smoke accept test to require a clean exit 0.
The typos hook auto-rewrites the deliberate single-edit misspellings in the `lenient` levenshtein/did-you-mean tests (src/mcp/lenient.rs) to `pattern`, which breaks them on every `prek run -a` gate. Allowlist them like `rela_path`.
Add `basemind checkpoint`: distil session text on stdin into a compact
{decisions, errors, files_changed} checkpoint a hook or agent can persist
or restore instead of re-reading the whole session. Decision and error
lines come from the text — reusing the safety.rs error markers plus a
conservative decision-marker set — while changed files come from the git
working tree, not regex-scraping. Any line carrying a credential is dropped
entirely before it can enter a field: a checkpoint is re-injected context
and must never leak a secret. Lists are deduplicated and capped with
explicit *_truncated flags (no silent caps).
Pure dependency-injected core (extract_checkpoint) with the git working-tree
fetch confined to the CLI runner.
Add `basemind detect-waste`: read a JSON-Lines tool-call log on stdin and flag wasteful token expenditure — redundant reads of the same path, repeated identical search/grep queries, and oversized full-file reads where an outline would suffice. Pure analysis; it never executes anything. Findings are credential-safe (any target embedding a secret is dropped before emission), deterministically ordered (by kind then target), and capped with an honest pre-truncation waste total so the headline never lies when the list is capped. Pure dependency-injected core (detect_waste) plus a lenient JSONL parse (parse_calls); the CLI runner only moves bytes. Completes the W6 behavioral- compression set (bash-output / delta / checkpoint / waste).
The compress tool now reports honest before/after token counts: a real o200k (gpt-4o) tokenizer via kreuzberg when built with the `documents` feature, and the `bytes/4` heuristic otherwise. New `src/mcp/tokens.rs` owns `count_tokens` + a `TOKENS_ARE_COUNTED` flag. `CompressResponse` renames `original/compressed_tokens_est` to `original/compressed_tokens` and adds `tokens_reduced` (the actually-removed count) and `tokens_counted` (real vs estimate). The structural path counts the original from the source on disk (fail-open to bytes/4); the prose path counts both texts directly. The real tokenizer downloads once and is cached, and stays on the explicit `compress` op — not any per-call hot path. Telemetry and budgeting remain on `bytes/4` pending the kreuzberg offline-tokenizer change.
`json_result` is used only by the memory tools (memory_put/get/list/search/ delete), all `#[cfg(feature = "memory")]`. The ungated import broke `--features documents` without `memory` with an unused-import error under `-D warnings`. Gate the import to match its use sites.
…emmem ESC scan Eliminate ~62.5k RelPath clones/call in proposals_mine by interning paths into a Vec<RelPath> + index map and keying co-change/cluster maps by usize. Bound the audit_scope_on_rescan Fjall scan with scan_cap = limit*8; `limit` now counts evaluated records, not all scanned. Replace the byte .contains(0x1b) ESC scan with memchr::memchr (SIMD).
Route the savings estimator through the real o200k tokenizer (count_tokens) when the full response text is in hand, instead of bytes/4, so telemetry reports honest token figures under the documents feature. Add estimate_from_text and a Cow-based result_text fast path. Extract record_call + result_text into a new helpers_telemetry.rs to keep helpers.rs under the 1000-line cap.
…(W7) Add serde aliases (query/pattern/needle/name/symbol/regex/q/search as apt) across the MCP param structs so a wrong-but-reasonable param name resolves instead of failing with an opaque -32602. Sweep covers search_symbols, workspace_grep, find_references/callers, dependents, call_graph, memory_*, git path/symbol params, compress expand, and governance audit. Tests assert each alias binds.
Drop to the freshly published rc.28 pin (kreuzberg + libheif + paddle-ocr + tesseract sub-crates). Build + full test suite green; config_schema's --features full snapshot skew is the known default-features artifact.
…r#19) `query outline` / `find-callers` / `call-graph` passed the raw user path straight into a RelPath, so absolute paths and `./`-prefixed paths missed the scanner's repo-relative index keys and returned a generic "file not indexed". Add `normalize_query_path` (strip repo-root prefix, drop leading `./`, collapse `.`/redundant separators, reject paths escaping the root) and apply it in the three path-bearing query subcommands. 8 exact-assertion unit tests. Closes Goldziher#19
…ldziher#21) `comms status` surfaced a bare "No such file or directory" when the daemon socket was absent. Map `NotFound`/`ConnectionRefused` from `UnixStream::connect` to a message naming that the daemon is not running and how to start it (`basemind comms start`); other connect errors keep their original context. Closes Goldziher#21
…dziher#25) `est_tokens_saved` was always 0 for ~half the tools because they sat in the `no_baseline` arm where baseline == actual. Tools that genuinely save tokens now carry honest, disclosed baselines: search_documents (~5x, full_document_read), list_files (2x, find_plus_filter), web_scrape/crawl/map (3x, manual_browse_paste). no_baseline is kept where there's no clean grep/read alternative (git, memory, cache, status). Exact-value tests under default; structural under documents. Closes Goldziher#25
The builtin hook keys were `polylint`/`polyfmt`, which poly 0.9.0 rejects with "unknown field" — the accepted names are `lint`/`fmt`. `poly lint .` failed to load the config until renamed.
Deterministic, LLM-free architecture overview built from real call edges. Ranks modules and symbols by graph centrality + git churn and surfaces circular-dependency clusters — never hallucinated, unlike LLM code maps. A whole-repo file-level call graph is built in memory at query time from the cached call sites (Fjall calls_by_path, or the in-RAM call cache in a read-only multi-session) — no new Fjall partition, no blob I/O. On it we compute deterministic signals: fan-in/out degree, fixed-iteration PageRank (damping 0.85, 20 iters, id-ordered — call-twice byte-identical), and iterative Tarjan SCC for cycle clusters. Three tiers via granularity: module (default, dirs collapsed at depth), file (base graph), and symbol (function hub ranking under a focus path-prefix). Ranked, Kneedle knee-cut, then bounded by the shared apply_budget token budgeter; edges emitted only between survivors. Optional include_churn overlay reuses hot_files. New kneedle knee-detection helper (Satopaa 2011, dependency-free). Wired per mcp-tool-conventions: types/tool-shim/helper/smoke/harden/README + CLI parity (query architecture-map). No new dependencies; no scanner changes.
Minor release — RELEASE_MINOR bumps 17 → 18, so the Fjall index and the content-addressed blob store wipe and rebuild from source on the next scan (standard minor-release migration). Bumps every shipped surface in lock-step via release:sync-version.
…an-in The repo-wide symbol tier ranked by raw name-based call count, so ubiquitous names dominated — on a real repo the top ~60 hubs were all `new` (fan_in ~9867), which misleads an agent routing on the result. Fan-in is now divided by the number of files defining that name (new RepoGraph::def_counts), so a genuine single-definition hub outranks a name that is merely everywhere. Raw `fan_in` stays reported verbatim. Also fixes a latent bug: fan_out was computed after the knee-cut, so the blended score never influenced selection or node ordering. Selection, knee-cut, and score now all key off the one specificity-weighted signal — emitted nodes are monotonic in score, matching the module/file tiers. Smoke test gains a decoy (high fan-in, multiply-defined) that must not outrank a true hub, plus a monotonic-score assertion. Verified on ../spikard: new/from/default no longer top the ranking; domain hubs (body/statusCode/read) surface instead.
Patch release — RELEASE_MINOR stays 18, no .basemind/ rebuild (query-side fix only). Bumps every shipped surface in lock-step via release:sync-version.
…ity labels QueryCursor was allocated per file (up to 3× with eager L2) despite being designed for reuse — now pooled in a thread-local alongside the parser pool (~3 fewer allocations per scanned file). capture_name, previously duplicated verbatim in l1.rs and l2.rs, moves to extract/mod.rs. entity_category_str returns Cow<'static, str> so the common (non-custom) NER categories don't allocate. Scanner output is byte-identical.
… rows
scan_paths unconditionally ran .replace('\\','/') on every path, allocating a
String even on Unix where no backslash exists — now cfg(windows)-guarded like
walk_candidates. The document-tier archive/binary extension check is an
AHashSet (O(1)) instead of a 40-element linear slice scan. build_doc_rows now
consumes the FileMapDoc and moves each chunk's text + embedding vector instead
of cloning them (a 768-dim f32 embedding is ~3 KB per chunk).
The posting value dropped a .to_vec() — lsm_tree::Slice implements From<[u8; N]>, so the 8-byte stack array is written without a heap copy. The forward-map value is now built in one sized allocation (Vec::with_capacity) instead of allocating a 4-byte Vec and reallocating on extend. On-disk bytes are identical (same big-endian doclen prefix + same rmp_serde term list).
…dependents diff_file cloned both (potentially multi-MB) file buffers only to preserve an is_some() existence check — now hoisted before the move. The blame/log/ commit-files disk-cache writers serialize from borrow-only structs (&[T]) instead of cloning the whole payload (wire format byte-identical: rmp_serde keys by field name). dependents drops a RelPath->PathBuf->RelPath round-trip over the entire file index (RelPath: AsRef<Path>). The watcher skips a backslash-normalizing String allocation on Unix.
…re_map bfs_callees precomputed a name->sites map once instead of re-scanning every indexed symbol per discovered callee (O(max_nodes x symbols) -> O(symbols + max_nodes)). architecture_map: callee_counts allocates a String only on a name's first sight rather than on all ~4M scanned call sites; the symbol-tier fan-out set skips the alloc on duplicate callees; PageRank reuses one accumulator buffer (20 allocs -> 1). Output stays byte-identical. Removed a dead last_emitted_key in find_implementations. DRY: the dual-backend call-site scan for_each_call_in_file now lives once in helpers_calls.rs, shared by architecture_map and the call-graph helpers (CallGraphSite gains Clone). CHANGELOG: document the whole perf sweep under Unreleased.
`pip install basemind` now ships a Hermes plugin, discovered via the `hermes_agent.plugins` entry point, that bundles the helper skills, slash commands, and agent-comms notifications (on_session_start + pre_llm_call). Tools still reach Hermes through its MCP config (`mcp_servers.basemind`) — a Hermes plugin cannot declare an MCP server. register(ctx) is stdlib-only and fail-open so it can never break the CLI that imports the same package. Adds pip-package/basemind/plugin.yaml as a version-synced release surface. Refs Goldziher#36.
/bm-statusline now writes a resolver as the statusLine command that re-resolves the newest installed statusline.sh at each render, instead of a version-pinned path that blanked the bar after an update. The bar also shows the running version (v<version>, full/compact tiers; BASEMIND_STATUSLINE_VERSION=0 opts out), read from the sibling plugin.json without spawning the binary.
Add Hermes install coverage to the README (per-tool MCP specifics) and the docs site (installation plugin + MCP tabs, index host list). Document architecture_map in the MCP tools reference and code-intelligence capability page — it shipped in 0.18.0 but was missing from the docs.
Minor release: index + blob schema bump (RELEASE_MINOR 18 → 19), so .basemind/ is wiped and rebuilt on the next scan. Ships the Hermes Agent plugin, the version-independent status line, the architecture_map docs, and the unreleased hot-path performance sweep.
inferno (transitive, all-features/all-targets only — the flamegraph rendering stack) is CDDL-1.0, inherited from the FlameGraph it ports. Add a per-crate license exception, matching the existing MPL-2.0 exceptions. Fixes the cargo-deny and poly validate CI jobs (poly runs cargo-deny).
normalize_absolute_outside_repo_passes_through_as_external_key asserts a leading-`/` external key round-trip. On Windows a bare `/other/...` input isn't absolute and drive-prefixed paths are rejected by normalize_absolute_components, so the case is inherently POSIX-shaped. Gate it to cfg(unix). Fixes the windows-latest comms/shells test job.
Invalid pointer dereference in crossbeam's `fmt::Pointer` impl for `Atomic`/ `Shared` (RUSTSEC-2026-0204), pulled transitively via rayon and inferno. Fixed upstream in 0.9.20 (crossbeam-rs/crossbeam#1276). Lockfile-only bump.
extra_roots keys external files by their leading-`/` absolute path; a Windows drive-prefixed path is rejected by normalize_absolute_components, so the suite is POSIX-shaped. Gate it to cfg(unix), matching the src/path.rs unit test. Fixes the remaining windows-latest comms/shells failures.
These three files had drifted from poly's format (single quotes, tab indent), failing the `poly fmt --check` step of the validate job. Formatting-only normalization — no semantic change.
External (extra_roots) files are keyed by their absolute path in forward-slash form. The scanner already emitted the Windows shape (`C:/…`), but the consuming side didn't: `is_external` only matched a leading `/`, and the query-side `normalize_absolute_components` rejected the drive `Prefix` entirely — so Windows external files were unqueryable and misclassified. Keep the drive prefix in the key (a shared `is_external_key` helper, drive detection Windows-gated), and normalize the doc-scope prefix match the same way. Unix keys are unchanged (byte-identical), so the index format is untouched. Ungate the path unit test and extra_roots_smoke suite to run on both platforms.
Patch: index + blob schema unchanged (RELEASE_MINOR stays 19). Fixes Windows scan.extra_roots keying and bumps crossbeam-epoch for RUSTSEC-2026-0204.
The persisted /bm-statusline resolver ranked cached statusline.sh copies by mtime (ls -dt), so an older version dir touched more recently won — showing a stale bar and the wrong version. Resolve the highest-versioned cache copy via sort -V (so it tracks the newest version the moment /plugin update installs it), fall back to the marketplace clone, and print a one-line hint instead of rendering blank when neither is found. Adds a smoke guard proving mtime picks the wrong version and sort -V the right one.
Ship a native Intel macOS (x86_64-apple-darwin) prebuilt binary — macOS is no longer Apple-Silicon-only. ort has no static x86_64-apple-darwin ONNX Runtime, so the Intel leg builds --features full,ort-dynamic (ort/load-dynamic; disable-linking wins over ort-bundled) and package-release.sh vendors Homebrew's libonnxruntime.dylib + its closure next to the binary with @loader_path install-names; ort resolves it relative to the executable, so ONNX loads with zero end-user setup. A gating CI smoke dlopens the vendored dylib on the Intel runner before upload. Homebrew formula ships both mac arches. Fix Apple Silicon under a Rosetta x86_64 shell: hw.optional.arm64 is masked under Rosetta, so the launcher/install.js/downloader.py fell through to the Intel branch and aborted. They now also probe sysctl.proc_translated and hand such shells the native arm64 binary. The launcher and pip downloader prune old per-version cache dirs so binaries stop accruing under ~/.cache/basemind. Refresh deps (cargo upgrade --incompatible): xberg 1.0.0-rc.11, fjall 3.1.6; arrow held at 58 until lancedb's transitive arrow moves.
Patch release — blob + index schema unchanged (RELEASE_MINOR stays 19), no .basemind wipe. Bundles the Intel macOS binary, Rosetta-aware installers, cache pruning, the statusline resolver fix, and the dependency refresh. Regenerates the config schema snapshot (crawler user_agent embeds the release version).
Scanning a fresh git worktree redid work the shared cache already held. Two reuse gaps are closed: - Code extraction: the "unchanged" fast-paths key off the per-view Fjall index, which is empty on a fresh worktree, so every file was re-parsed even though its content-addressed L1/L2 blob already existed in the shared (main-worktree) blob store. process_file now deserializes the cached frame when present instead of re-running tree-sitter. - git-history index: it is derived entirely from the shared .git object database, yet each worktree rebuilt its own. A linked worktree now resolves the index to the main worktree's .basemind (mirroring resolve_blobs_dir), so the rebuild is paid once. On a large monorepo a cold worktree scan drops from ~181s to ~25s (~7x) with ~2.6x less peak RSS; 67700/67702 files reused. Adds a `reused` scan-summary counter and regression tests for both paths.
…ngs off by default Move config to <root>/basemind.toml (committable, survives .basemind wipes); the legacy .basemind/basemind.toml is still read for back-compat. `basemind init` now scaffolds a fully-documented root config and ensures `.basemind/` is gitignored. Scan filtering: - Always-applied exclude floor (node_modules, .venv, __pycache__, target, dist/build/out, bazel-*, vendored/cache/editor dirs, …) added on top of `scan.exclude` — narrowing the user list never drops a floor entry. - New `scan.follow_symlinks` (default false), threaded through the walker; the floor's bazel-* entries keep symlink-following from pulling in generated trees. Embeddings: - `code_search.embed` defaults to false. A general-English model embeds code weakly and the BM25 keyword lane already covers the NL->symbol query over the same text, so code embedding isn't worth the ONNX download + ORT pass + vector store. Code is still chunked + keyword-indexed; docs/images still embed. - New `embed_exclude` globs (code + documents): skip embedding a file while keeping it indexed. - New `documents.extract_archives` (default false): opt into xberg archive extraction. Archive vs true-binary denylists are split so the toggle only relaxes archives. README config section rewritten for the root location + new knobs; schema regenerated.
The per-file embedding cache keyed on vector dim only, but `balanced` and `multilingual` both produce 768-dim vectors — so switching between them reused the old model's cached vectors into the (correctly wiped) LanceDB table, leaving stale-model embeddings. Cache-reuse now also requires the embedding model/preset to match: - CodeChunkBlob gains a `#[serde(default)] embedding_model` (blob-compatible; pre-existing blobs deserialize empty -> cache miss -> re-embed). No schema bump. - Both the code and document cache-reuse predicates compare the model. LanceStore::open already wipes+rebuilds the vector dir on any (dim, model, schema) change, so a preset change now cleanly drops the old vectors AND re-embeds under the new model.
naamanhirschfeld-armis
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related improvements to basemind's scan/index behavior, driven by debugging runaway CPU + disk on a large (armis) monorepo.
1. Reuse the shared cache across git worktrees (
165565d)A fresh worktree re-did work the shared cache already held. Now:
.git) instead of rebuilt per worktree.Cold worktree scan on the monorepo dropped ~181s → ~25s (~7×), 67700/67702 files reused, peak RSS 3.07G → 1.19G.
2. Root config, scan-filter floor + symlinks, code embeddings off by default (
4912635)<root>/basemind.toml(committable, survives.basemind/wipes); legacy.basemind/basemind.tomlstill read.basemind initscaffolds a fully-documented root config and ensures.basemind/is gitignored.__pycache__, target, dist/build,bazel-*, vendored/cache/editor dirs, …) added on top ofscan.exclude— narrowing the user list never drops a floor entry.scan.follow_symlinks(default false), threaded through the walker.code_search.embeddefaults tofalse. A general-English model embeds code weakly, and the BM25 keyword lane already covers the NL→symbol query over the same text — so code embedding isn't worth the ONNX download + ORT pass + vector store. Code is still chunked + keyword-indexed; docs/images still embed.embed_excludeglobs (code + documents) skip embedding a file while keeping it indexed;documents.extract_archives(default off) opts into xberg archive extraction (archive vs true-binary denylists split so the toggle only relaxes archives).3. Safe embedding-preset changes (
281c435)LanceStorealready wipes+rebuilds the vector dir on any(dim, model, schema)change. The per-file embedding cache, however, keyed on dim only — so switching between two same-dim presets (balanced↔multilingual, both 768) reused stale-model vectors. Cache-reuse now also requires the model to match (both tiers), so a preset change cleanly drops the old vectors and re-embeds under the new model.CodeChunkBlobgains a#[serde(default)] embedding_model(blob-compatible, no schema bump).Tests / verification
cargo fmt,clippy -D warnings, and fullcargo test --workspacegreen. New tests for worktree reuse, root/legacy config discovery, exclude-floor,embed_exclude, archive toggle,init+.gitignore, and same-dim preset-model reuse. (The--features fulllink fails locally on vendoredxberg-tesseract/libheifnative symbols — an env issue, unrelated; feature-gated code verified via clippy.)