Skip to content

sdkks/symbol-index

BranchesTags

Repository files navigation

symbol-index

MCP server for AI agents to explore large codebases via symbol-level search and semantic similarity. Supports 305 languages via tree-sitter-language-pack.

Install

# Homebrew (macOS — Apple Silicon)
brew tap sdkks/tap && brew install sdkks/tap/symbol-index

# Cargo (any platform — requires Rust 1.86+)
cargo install --git https://github.com/sdkks/symbol-index
# macOS: if the build fails with a protobuf-related linker error, run:
#   brew install protobuf
# then retry the cargo install command.

# From source
git clone https://github.com/sdkks/symbol-index
cd symbol-index
make install    # builds release and installs to ~/.cargo/bin

Quick start

# Index a repository (cd first so project ID is stable regardless of path form)
cd /path/to/your/repo && symbol-index index .

# Search for symbols by name (case-insensitive, FTS5)
symbol-index search "handle_request"

# Search by meaning (vector similarity)
symbol-index search-semantic "function that handles HTTP requests"

# Filter by kind (struct, fn, class, etc. — aliases accepted)
symbol-index search "store" --kind struct

# Look up a specific symbol (dot notation, not ::)
symbol-index get "MyStruct.method_name"

# Check index status
symbol-index status

Modes of operation

Invocation Mode Description
symbol-index (no args) MCP server Listens on stdio for MCP JSON-RPC requests from AI agents
symbol-index index <path> CLI One-shot indexing with JSON output
symbol-index search <query> CLI Search symbols by name or signature (FTS5)
symbol-index search-semantic <query> CLI Search symbols by meaning (vector similarity)
symbol-index get <name> CLI Look up a symbol by qualified name
symbol-index status CLI Show index statistics
symbol-index flush CLI Clear all index data for current project

MCP setup

Wire it into your MCP client by adding to mcp.json:

{
  "mcpServers": {
    "symbol-index": {
      "command": "symbol-index",
      "args": []
    }
  }
}

Or install with Claude Code:

# Use full path to binary if it's not in your $PATH
# Validate which command `which symbol-index`
claude mcp add symbol-index -- symbol-index # project scope
claude mcp add symbol-index --scope user -- symbol-index # user scope (global)
claude mcp add symbol-index --scope user -- /some/path/symbol-index # user scope if binary not in $PATH

CLI subcommands

symbol-index index <path>          Index a repository (auto: incremental if previously indexed, full otherwise)
symbol-index index <path> --incremental
                                   Index only changed files (Merkle tree)
symbol-index index <path> --diff   Index only git-changed files (since last index)
symbol-index index <path> --force  Force full re-index (override auto-default)

symbol-index search <query>        Search symbols by name/signature (FTS5, case-insensitive)
  --kind <type>                    Filter by kind (function, struct, class, etc.)
  --language <lang>                Filter by language (rust, typescript, python, etc.)
  --limit <N>                      Max results (default 20)
  --project <id>                   Filter by project ID
  --format <json|pretty>           Output format (default pretty)

symbol-index search-semantic <query>
                                   Search by meaning (vector similarity)
  --limit <N>                      Max results (default 10)
  --threshold <F>                  Min similarity 0.0-1.0 (default 0.3)
  --project <id>                   Filter by project ID
  --format <json|pretty>           Output format (default pretty)

symbol-index get <qualified_name>  Look up a symbol by qualified name (dot notation)
  --project <id>                   Filter by project ID
  --format <json|pretty>           Output format (default pretty)

symbol-index status                Show index statistics
  --path <path>                    Repository path (default: current directory)
  --format <json|pretty>           Output format (default pretty)

symbol-index flush                 Clear index data (SQLite + LanceDB) for current project
symbol-index --version             Show version and git SHA
symbol-index help                  Show help message

Qualified name notation

Qualified names always use dot notation — not language-specific separators. This is a deliberate design choice: since symbol-index supports 305 languages (each with its own path separator), dot notation provides a single universal syntax.

Language Native syntax symbol-index QN
Rust std::io::Read std.io.Read
Python os.path.join os.path.join
Java java.util.List java.util.List
TypeScript React.useState React.useState

Methods use dot notation from their parent type:

# Rust: impl SqliteStore { fn open(...) }
symbol-index get "SqliteStore.open"

# Python: class MyClass { def method(self) }
symbol-index get "MyClass.method"

# TypeScript: class UserService { login() }
symbol-index get "UserService.login"

Top-level symbols (functions, classes not inside a module) use bare names:

symbol-index get "discover_files"
symbol-index get "main"

Why not ::? Rust uses ::, Python uses ., Java uses . — there's no universal separator. Rather than requiring language-specific knowledge to look up symbols, dot notation works identically for all 305 languages. The search command (case-insensitive FTS5) is the recommended way to discover exact qualified names.

Kind filter aliases

The --kind filter accepts common aliases — use whatever feels natural:

Input Maps to Examples
struct, impl, type type Rust structs, impl blocks
fn, func, function function Top-level functions
class class Python/Java/TS classes
method, meth method Methods on types
trait, interface, iface interface Rust traits, TS interfaces
enum enum Enumerations
mod, module module Module declarations
const, constant constant Constants
var, variable variable Variables
field, member field Struct/class fields

MCP tools

When running in MCP mode, 16 tools are exposed to the AI agent:

Tool Description
fff_find_files FFF-backed fuzzy file/path discovery across repository files
fff_grep FFF-backed broad text search across source, docs, config, tests, and prose
fff_multi_grep FFF-backed multi-query text search with grouped and annotated results
search_symbol FTS5 text search by name, signature, or docstring
search_semantic Vector similarity search with natural language queries
explore Unified search: parallel FTS5 + semantic, merged with cross-path boosting
get_symbol Exact lookup by qualified name
get_related Cross-linked discovery: same-file + same-module + semantic neighbors
get_callers Call graph: find all callers of a symbol (impact analysis)
get_callees Call graph: find all symbols called by a symbol (dependency tracking)
trace_path Call graph: BFS path finding between two symbols
index_repo Full/incremental/diff indexing (auto-detects mode, embeddings always generated)
flush_repo Delete all index data (SQLite + LanceDB) for a fresh start
index_status Index stats: symbol count, embedded count, file count, languages, model
status Alias for index_status
version Server version, git SHA, and build target

Use FFF tools for broad repository discovery when the symbol index is stale, indexing, not indexed, unavailable, or when symbol search returns poor first results. They are also the right first choice for docs, ADRs, configuration, prose, scripts, tests, examples, fixtures, and other non-symbol content. Use symbol tools for precise structured navigation once candidate symbols or files are known. See FFF Discovery Tools for repo_path, state, limits, and safety defaults.

Recommended workflow

# First run — full index (embeddings auto-generated for semantic search)
cd /path/to/repo && symbol-index index .

# Daily development — fast incremental updates (auto-detects existing index)
symbol-index index /path/to/repo --diff

# After major refactoring — flush stale data, rebuild fresh
symbol-index flush
symbol-index index /path/to/repo --force

Large codebases

For repositories with 100,000+ symbols, always index from the CLI — the MCP index_repo tool may timeout. The CLI has no time limit and shows real-time progress.

# Build the index manually — the agent will use it once it's ready
cd /path/to/large-repo && symbol-index index .

Once indexed, you can verify with symbol-index status. The MCP server serves any existing index — the agent can immediately search and explore without re-indexing.

In MCP mode, the agent can call flush_repo then index_repo with mode full to get a fully fresh view of smaller codebases after major changes.

Indexing modes

  • Full (--mode full): Parses every file, builds complete index. Use for first run.
  • Incremental (--incremental): Compares file mtime/size against stored metadata. Only re-parses changed files.
  • Diff (--diff): Git-aware. Compares last indexed commit against HEAD. Includes committed changes, staged changes, and untracked files. Fastest for active development.

How it works

flowchart TB
    subgraph Input["📂 Codebase"]
        src["Source Files<br/>305 languages"]
    end

    subgraph Index["⚙️ symbol-index"]
        direction TB
        parse["tree-sitter Parse<br/>Extract symbols, signatures, docstrings"]
        store["SQLite FTS5<br/>Full-text search + metadata"]
        vectors["LanceDB Vectors<br/>768-dim embeddings"]
        parse --> store
        parse --> vectors
    end

    subgraph Query["🔍 AI Agent"]
        direction TB
        q1["search_symbol<br/>Find by name/signature"]
        q2["search_semantic<br/>Find by meaning"]
        q3["get_related<br/>Cross-linked discovery"]
    end

    subgraph Benefit["💰 Token Savings"]
        direction TB
        t1["Skip reading 1000s of files"]
        t2["Jump directly to relevant code"]
        t3["Understand codebase structure<br/>without exploration"]
    end

    src -->|"Index once"| Index
    Index -->|"Query many"| Query
    Query --> Benefit

    style Input fill:#e1f5fe
    style Index fill:#fff3e0
    style Query fill:#e8f5e9
    style Benefit fill:#fce4ec
Loading

Why this saves tokens and time

Without symbol-index, an AI agent exploring a large codebase must:

  1. Read entire files to find relevant functions
  2. Search through grep results with no semantic understanding
  3. Spend 10-50k tokens on exploration before doing useful work

With symbol-index:

  1. Index once — parse the entire codebase into a compact symbol database
  2. Query precisely — find exact symbols in milliseconds, not minutes
  3. Semantic search — describe what you need in natural language
  4. Cross-linking — discover related symbols without reading source files

Result: 90%+ reduction in tokens spent on codebase exploration.

Architecture

                          symbol-index
                               |
               +---------------+---------------+
               |               |               |
         tree-sitter      SQLite FTS5     LanceDB vectors
         (parsing)        (text search)   (semantic search)
               |               |               |
         305 languages    BM25 ranking    cosine similarity
               |               |               |
               +-------+-------+-------+-------+
                       |               |
                  symbol lookup    related symbols
                  (exact match)   (cross-linked)
  1. Parse: tree-sitter-language-pack extracts symbols (functions, classes, methods, types, etc.) from source files
  2. Store: Symbols go into SQLite with FTS5 full-text search; embeddings go into LanceDB for vector search
  3. Search: Agents can search by name (FTS5), by meaning (vector similarity), or find related symbols (cross-linked)
  4. Index: Supports full, incremental (mtime/size), and diff (git-aware) indexing modes

Cross-linking: SQLite + LanceDB

SQLite and LanceDB are linked via a shared lance_id ({project_id}:{file_path}:{qualified_name}). This enables the get_related tool to combine both stores on the fly:

  • Textual path — SQLite finds symbols in the same file or module (no LanceDB needed)
  • Semantic path — LanceDB finds embedding neighbors by cosine similarity, then resolves back to full symbol records in SQLite via lance_id

There is no stored graph — relationships are computed fresh on each query. This means the cross-links always reflect the current index state without needing maintenance.

Configuration

Embedding model cache

The first time you index, symbol-index downloads the jina-embeddings-v2-base-code ONNX model (~350 MB). Embeddings are always generated automatically. By default the model is stored at ~/.cache/fastembed so all projects share a single copy.

To store the model elsewhere, set the FASTEMBED_CACHE_DIR environment variable:

export FASTEMBED_CACHE_DIR="/path/to/models"

Troubleshooting

TLS / model download errors

symbol-index only needs network access once — to download the embedding model (~350 MB ONNX file) on first indexing run. After that, it works entirely offline.

If you see TLS errors during model download (e.g. UnknownIssuer), your machine is likely behind a corporate TLS-inspecting proxy (Netskope, Zscaler, etc.) that re-signs HTTPS traffic with a root CA not visible to fastembed's HTTP layer.

The workaround is to route model downloads through a local HuggingFace mirror that proxies over plain HTTP. See docs/tls-ca-issue.md for the full guide.

Supported languages

symbol-index uses tree-sitter-language-pack which provides grammars for 305 languages. The file walker currently recognizes these extensions for indexing:

Systems & compiled: Rust (.rs), C (.c, .h), C++ (.cpp, .cc, .cxx, .hpp), Go (.go), Zig (.zig), Swift (.swift), Assembly (.asm, .s)

JVM: Java (.java), Kotlin (.kt, .kts), Scala (.scala, .sc)

Scripting & web: Python (.py, .pyi), JavaScript (.js, .jsx, .mjs, .cjs), TypeScript (.ts, .tsx, .mts, .cts), PHP (.php), Ruby (.rb, .erb), Lua (.lua), Perl (.pl, .pm), Elixir (.ex, .exs), Dart (.dart)

Functional: Haskell (.hs, .lhs), OCaml (.ml, .mli)

Data & scientific: R (.r), Julia (.jl), SQL (.sql)

Shell & config: Bash/Zsh (.sh, .bash, .zsh), TOML (.toml), YAML (.yaml, .yml), JSON (.json, .jsonc), XML (.xml, .svg)

Web & markup: HTML (.html, .htm), CSS (.css, .scss, .sass, .less), Markdown (.md, .mdx)

Build & infra: Dockerfile (.dockerfile), Makefile (.mk), Protocol Buffers (.proto)

Also supported but not yet mapped: The underlying parser handles 305 languages — if a language isn't listed above, its files are silently skipped during indexing. To add support for a new extension, edit detect_language() in src/indexer/walker.rs.

Conventional commits

This project follows Conventional Commits for automated versioning. Every push to main triggers a semantic version bump based on commit messages:

Commit type Release Example
feat!: or BREAKING CHANGE: footer major feat!: drop support for Python 3.8
feat: minor feat: add incremental embedding support
fix:, perf:, refactor:, revert: patch fix: canonicalize repo path in index command
docs:, style:, chore:, ci:, test:, build: none docs: update install instructions

Breaking changes are detected via either the ! notation (feat!:, fix(api)!:) or a BREAKING CHANGE: footer in the commit body.

Observability

Performance profiling

The profiling Cargo profile inherits release optimization but keeps full debug info and symbols.

cargo build --profile profiling

Two tools are documented below. They produce different artifacts and serve different purposes — pick based on what you need.

Instruments.app (macOS) — comprehensive timeline profiler

Uses xctrace under the hood. Produces a .trace file that opens in Instruments.app with a full-featured timeline view. This is the most comprehensive option: you can inspect CPU usage over time, drill into specific time ranges, view per-thread activity, and use multiple analysis modes (Time Profiler, Allocations, Leaks, System Trace).

cargo install cargo-instruments
cargo instruments --profile profiling -t "Time Profiler" --bin symbol-index -- index "$PWD"# Trace opens automatically in Instruments.app
# Use File > Export to save a flamegraph from the selected time range

flamegraph (Linux, macOS) — single SVG flame graph

Wraps the platform profiler and produces a single interactive SVG. Shows aggregated stack traces (width = time spent) without a time axis. Best for quick hotspot identification and sharing results — the SVG opens in any browser with click-to-zoom.

# Linux (uses perf)
cargo install flamegraph || cargo install --git https://github.com/flamegraph-rs/flamegraph.git
cargo flamegraph --profile profiling --bin symbol-index -- index /path/to/repo
# macOS (uses xctrace under the hood, outputs SVG directly)
cargo install flamegraph || cargo install --git https://github.com/flamegraph-rs/flamegraph.git
cargo flamegraph --profile profiling --bin symbol-index -- index "$PWD"```

#### Which one?

| | Instruments.app | flamegraph |
|---|---|---|
| **Output** | `.trace` file | `.svg` file |
| **View** | Timeline + call trees | Aggregated flame graph |
| **Time axis** | Yes — zoom into spikes | No — all samples merged |
| **Thread analysis** | Per-thread breakdown | Merged across threads |
| **Other modes** | Allocations, Leaks, GPU, etc. | CPU time only |
| **Sharing** | Requires Instruments.app | Any browser |
| **Best for** | Deep investigations | Quick hotspot checks |

The profiling profile settings:
- `debug = 2` — full DWARF debug info
- `strip = "none"` — all symbols preserved
- `lto = false` — no link-time optimization (cleaner call stacks)
- `opt-level = 2` — near-release speed, not size-optimized

## Build

```bash
cargo build --release    # release build (stripped, LTO optimized)
cargo test               # run tests
make lint                # format + clippy
make fix                 # auto-fix formatting and clippy suggestions
make install             # install to ~/.cargo/bin
make coverage            # run coverage (requires cargo-llvm-cov)

License

MIT

About

Index your very large codebases to significantly reduce token in/out and context window cost

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 36

v0.22.0 Latest
Jul 6, 2026
+ 35 releases

Packages

 
 
 

Contributors

Languages