Local search rework proposal

[adamtagscherer]

Search Functionality Enhancement Proposal

Overview

This document proposes enhancements to the local search functionality in the Directory, addressing limitations in the current implementation and enabling more powerful, flexible queries for AI agents and CLI users.

Current State

The search functionality is currently accessible from two interfaces:

1. CLI: cli/cmd/search/search.go
2. MCP Tool: mcp/prompts/search_records.go

Current Capabilities

• Wildcard matching: Supports *, ?, and [abc] patterns
• Multiple field types: name, version, skill-id, skill-name, locator, module, domain-id, domain-name
• Multi-value filters: Can specify multiple values per field (implicit OR within same field, implicit AND between different fields)
• Pagination: Supports limit and offset

Current Limitations

1. No boolean operators: Cannot combine conditions with AND, OR, NOT, ... across different fields
2. No comparison operators: Cannot perform range queries (e.g., version >0.1.0)
3. Limited field coverage: Missing description, created_at, ... field search
4. No query composition: Cannot express complex queries like "(skill A AND domain B) OR (skill C AND version >2.0)"

Example Current Usage

# Find agents with Python skill AND in education domain
dirctl search --skill "*python*" --domain "*education*"

# Multiple names are OR'd together
dirctl search --name "web*" --name "api*"

Problem Statement

AI agents need to perform more sophisticated searches to accurately discover relevant records. For example:

• "Find Python agents in the education domain with version >= 2.0"
• "Find agents that do image processing OR video processing, but NOT in the healthcare domain"
• "Find MCP agents with description containing 'database' and version starting with v1"

The current system cannot express these queries without returning false positives or requiring post-processing.

---

Proposed Solutions

Option 1: Query Expression Language with CLI Flags

Concept

Introduce boolean operators as CLI flags and add new field types with comparison operators. Use structured flags to build an expression tree on the backend.

I opened a PR to test how can we parse and Expression tree on the backend: #710

CLI Interface

# AND operator (requires all conditions)
dirctl search \
  --and \
  --skill "*python*" \
  --domain "*education*" \
  --version-gte "v2.0.0"

# OR operator (any condition matches)
dirctl search \
  --or \
  --skill "*image*" \
  --skill "*video*"

# NOT operator (exclude matches)
dirctl search \
  --skill "*python*" \
  --not --domain "healthcare"

# Complex nested queries using groups
dirctl search \
  --group-and \
    --skill "*python*" \
    --domain "*education*" \
  --end-group \
  --or \
  --group-and \
    --skill "*javascript*" \
    --domain "*web*" \
  --end-group

# New comparison operators for version
dirctl search \
  --name "my-agent" \
  --version-gte "v1.0.0" \
  --version-lt "v2.0.0"

# New description field
dirctl search \
  --description "*database*" \
  --and \
  --module "*mcp*"

CLI Implementation (cli/cmd/search/):

1. Add new flags: --and, --or, --not, --group-and, --group-or, --end-group, ...
2. Add version comparison flags: --version-gt, --version-gte, --version-lt, --version-lte, ...
3. Add description flag: --description, ....
4. Build expression tree from flags
5. Convert tree to protobuf QueryExpression

Pros

• Strongly typed: Type-safe, validated at protobuf level
• Explicit semantics: Clear meaning, no ambiguity
• Extensible: Easy to add new operators and fields
• CLI-friendly: Each flag has clear meaning
• No parsing complexity: Direct mapping from flags to structure

Cons

• Verbose CLI: Complex queries require many flags
• Cognitive load: Users must understand operator precedence and grouping
• Flag explosion: Many new flags to maintain
• Group syntax: --group-and ... --end-group can be confusing
• Not human-readable: Harder for agents to generate dynamically

---

Option 2: String-Based Query Language (Parsing Approach)

Concept

Define a simple query language syntax that can be expressed as a single string. Parse the string on the backend into an expression tree.

CLI Interface

# Simple AND query
dirctl search --query 'skill *python* AND domain *education* AND version >=v2.0.0'

# OR query
dirctl search --query 'skill *image* OR skill *video*'

# NOT query
dirctl search --query 'skill *python* NOT domain healthcare'

# Complex nested query with parentheses
dirctl search --query '(skill *python* AND domain *education*) OR (skill *javascript* AND domain *web*)'

# Comparison operators
dirctl search --query 'name my-agent AND version >=v1.0.0 AND version <v2.0.0'

# New description field
dirctl search --query 'description ~database AND module *mcp*'

# Multiple conditions
dirctl search --query 'name web* AND (skill *api* OR skill *rest*) AND version >=v2'

Pros

• Concise CLI: Single string for entire query
• Human-readable: Easier to read and understand
• Familiar syntax: Similar to search engines, SQL, etc.
• Composable: Easy to programmatically generate queries
• Agent-friendly: LLMs can easily generate query strings
• Copy-pasteable: Share queries easily

Cons

• Parsing complexity: Need robust parser with error handling
• Escaping issues: Special characters in values need escaping
• Syntax errors: Users can write invalid queries
• Learning curve: Users must learn query syntax
• Quoting complexity: Spaces and special chars require quotes

---

Option 3: LLM-Based Natural Language Query

Concept

Accept free-text queries and use an LLM on the backend to translate them into structured database queries. The LLM has access to the database schema and generates the appropriate query.

CLI Interface

# Natural language queries
dirctl search --nl "Find Python agents in the education domain with version 2.0 or higher"

dirctl search --nl "Show me agents that can process images or videos but not in healthcare"

dirctl search --nl "MCP agents with database functionality released after version 1.5"

dirctl search --nl "agents for natural language processing in finance"

Pseudocode

func main() {
    // Initialize database and tool handlers
    db := ConnectToDatabase()
    
    toolHandlers := map[string]ToolHandler{
        "tableNames":       {Tool: TableNamesTool(), Handler: func(params string) (any, error) {
            return db.GetAllTableNames()
        }},
        "tableDescription": {Tool: TableDescriptionTool(), Handler: func(params string) (any, error) {
            var p struct{ TableName string }
            json.Unmarshal([]byte(params), &p)
            return db.DescribeTable(p.TableName)
        }},
        "runSQLQuery":      {Tool: RunSQLQueryTool(), Handler: func(params string) (any, error) {
            var p struct{ Query string }
            json.Unmarshal([]byte(params), &p)
            return db.ExecuteQuery(p.Query)
        }},
        "schemaOverview":   {Tool: SchemaOverviewTool(), Handler: func(params string) (any, error) {
            return db.GetSchemaOverview()
        }},
    }
    
    // Extract tool definitions for LLM
    tools := make([]Tool, 0, len(toolHandlers))
    for _, handler := range toolHandlers {
        tools = append(tools, handler.Tool)
    }
    
    // Initialize LLM client and conversation
    client := openai.NewClient(apiKey)
    dialogue := []Message{
        {Role: "system", Content: "You are a database expert. Use available tools to answer queries."},
        {Role: "user", Content: "What agents have cloud technology skills?"},
    }
    
    // Main conversation loop
    for {
        response := client.CreateChatCompletion(ctx, Request{
            Messages: dialogue,
            Tools:    tools,
        })
        
        dialogue = append(dialogue, response.Message)
        
        if len(response.Message.ToolCalls) == 0 {
            fmt.Printf("Final answer: %s\n", response.Message.Content)
            return
        }
        
        // Execute all requested tools
        for _, toolCall := range response.Message.ToolCalls {
            handler := toolHandlers[toolCall.Function.Name]
            result, err := handler.Handler(toolCall.Function.Arguments)
            
            dialogue = append(dialogue, Message{
                Role:       "tool",
                Content:    fmt.Sprintf("%v", result),
                ToolCallID: toolCall.ID,
            })
        }
    }
}

Pros

• Best UX: Most intuitive, no syntax to learn
• Agent-friendly: Perfect for LLM-based agents (MCP use case)
• Flexible: Handles ambiguous or complex queries
• Future-proof: Can improve as LLMs improve
• Fallback friendly: Can fall back to existing solution
• Easy to maintain: No need to change business logic after DB change

Cons

• Cost: API calls cost money (unless local LLM)
• Non-deterministic: Same query might produce different results

---

Recommendation

I think that as a first step we should keep option 1 so that the feature set stays the same, and implement option 3. With option 3 we will have a solution that can extract the DB schema and create arbitrary queries based on what seems appropriate. We won't have to change the business logic as the DB changes.

[tkircsi]

Option 3 provides a critical architectural benefit: it decouples the query interface from the underlying data model. As the schema evolves (new fields, relationships, or database migrations), the business logic remains unchanged. The LLM acts as an adaptive translation layer that can:

• Dynamically discover schema changes via tool calls (tableDescription, schemaOverview)
• Generate appropriate queries without hardcoded field mappings
• Handle schema versioning transparently

This is fundamentally different from Options 1 & 2, which require code changes whenever new searchable fields are added or the data model evolves.

Reduced Technical Debt

Option 1 leads to flag proliferation. As the schema grows, you'd need:

• New flags for each field + operator combination
• Maintenance of flag parsing logic
• CLI documentation updates
• Breaking changes when operators change

Option 2 requires maintaining a query parser with its own grammar, lexer, and error handling—essentially building a mini query language. This introduces:

• Parser maintenance burden
• Edge case handling (escaping, quoting, precedence)
• Version compatibility concerns
• Documentation of query syntax

Option 3 externalizes this complexity to the LLM, which already handles natural language parsing at scale.

Addressing Non-Determinism

Acceptable Variance in Practice

The non-determinism concern is valid but overestimated in impact:

1. Bounded output space: The LLM generates SQL queries against a fixed schema. The variance is in query construction, not arbitrary outputs.
2. Tool-constrained behavior: The LLM can only call predefined tools (runSQLQuery), preventing hallucination beyond the schema.
3. Deterministic execution: Once the SQL is generated, the database engine provides deterministic results.

MCP Use Case Alignment

The MCP (Model Context Protocol) use case is critical context. MCP is designed for AI agents to interact with tools. Option 3 is native to this paradigm:

• MCP agents naturally generate natural language queries
• Option 1/2 requires agents to learn flag syntax or query DSL (brittle)
• Option 3 allows agents to pass queries directly (robust)

Implementation Risk Mitigation

Security

• SQL validation: Parse generated SQL AST to ensure only SELECT statements
• Table allowlisting: Restrict queries to specific allowed tables
• Read-only access: Prevent any destructive operations

Option 3 is the architecturally superior choice for a system targeting AI agent interactions. The non-determinism trade-off is acceptable.

Key Benefits Summary

1. Zero maintenance on schema changes
2. Optimal UX for human and AI agents
3. Future-proof architecture
4. Native alignment with MCP use case
5. Acceptable costs for developer tooling

Recommendation: Implement Option 3 with query validation, caching, and optional structured fallback for deterministic requirements.

[arpad-csepi]

Option 2 looks better IMO than Option 1 as more complex query might not be able to work (depends on how cobra parses the flags).
Also Option 2 can go together well with Option 3 later as we can use LLM to generate the regex like search query, validate that with a tool and show the user before proceed.

[lgecse]

Here is my weighted preference list, starting with the option I believe is strongest:

• Option 2, because it's deterministic
• Option 3, because of it's UX
• Option 1 (I don't like this much to be honest)

I'd do both Option 2 and 3.

[ramizpolic]

I agree with @lgecse on the order of preference. I think supporting option 2 alone is enough. Once the query syntax is supported and documented, it is automatically supported by the MCP server to enable option 3 (meaning, we dont have to do anything/much to support option 3).

Regarding @tkircsi comment on

Option 2 requires maintaining a query parser with its own grammar, lexer, and error handling—essentially building a mini query language

is only partially true. We can use libraries such as https://github.com/alecthomas/participle to only define the query grammar, which can be natively mapped to whatever the backend db query builder. This simplifies the implementation greatly and reduces maintenance to only the grammar itself.

I would suggest Option 2 with a caveat of integration with existing tools (query builders, etc) and avoid implementation on our end. This should be fairly straightforward iff we select the proper tooling to support it.

[tkircsi]

I can go with Option 2, but my personal opinion is Option 3 would be a better and more modern choice. The participle suggestion makes Option 2 less complex than Option 1, but it doesn't make it simple - it's still significant engineering(i.e.: grammar design, query translation layer, handle operator precedence, support wildcards, etc.... ) and ongoing maintenance.

[adamtagscherer]

We decided to not rework the current search functionality, only add specific fields to it: created_at, authors, schema_version, and module_id. Besides that, I will implement greater than, lesser than operators on some fields: version, schema_version and created_at.