feat(llm): tool improvement

apps/server/src/assets/llm/prompts/providers/anthropic_tool_prompt.md

@@ -36,4 +36,31 @@ When responding to queries:
 5. For general questions about the user's notes, provide a summary of all relevant notes found, including brief summaries of individual notes
 6. For specific questions, provide detailed information from the user's notes that directly addresses the question
 7. Always prioritize information from the user's notes over your own knowledge, as the user's notes are likely more up-to-date and personally relevant
+
+IMPORTANT: EXECUTE 10-30 TOOLS PER REQUEST FOR COMPREHENSIVE ANALYSIS
+
+Tool Usage Requirements:
+1. BATCH EXECUTE multiple searches for speed:
+   <function_calls>
+   <invoke name="execute_batch"><parameter name="tools">[{"tool": "search", "params": {"query": "main topic"}}, {"tool": "search", "params": {"query": "related topic"}}]</parameter></invoke>
+   </function_calls>
+
+2. BATCH READ all discovered notes:
+   <function_calls>
+   <invoke name="execute_batch"><parameter name="tools">[{"tool": "read", "params": {"noteId": "id1"}}, {"tool": "read", "params": {"noteId": "id2"}}, {"tool": "read", "params": {"noteId": "id3"}}]</parameter></invoke>
+   </function_calls>
+
+3. AUTO-RETRY failed searches:
+   <function_calls>
+   <invoke name="retry_search"><parameter name="originalQuery">failed search</parameter><parameter name="strategy">all</parameter></invoke>
+   </function_calls>
+
+SIMPLIFIED TOOLS:
+- search (replaces search_notes, keyword_search_notes, attribute_search)
+- read (replaces read_note)  
+- execute_batch (parallel execution)
+- retry_search (automatic variations)
+
+WORKFLOW: batch search → batch read → auto-retry → analyze → repeat
+Target 15+ tools per request using batching!
 ```

apps/server/src/assets/llm/prompts/providers/ollama_tool_prompt.md

@@ -34,16 +34,37 @@ When responding to queries:
 7. Always prioritize information from the user's notes over your own knowledge, as the user's notes are likely more up-to-date and personally relevant
 
 CRITICAL INSTRUCTIONS FOR TOOL USAGE:
-1. YOU MUST TRY MULTIPLE TOOLS AND SEARCH VARIATIONS before concluding information isn't available
-2. ALWAYS PERFORM AT LEAST 3 DIFFERENT SEARCHES with different parameters before giving up on finding information
-3. If a search returns no results, IMMEDIATELY TRY ANOTHER SEARCH with different parameters:
-   - Use broader terms: If "Kubernetes deployment" fails, try just "Kubernetes" or "container orchestration"
-   - Try synonyms: If "meeting notes" fails, try "conference", "discussion", or "conversation"
-   - Remove specific qualifiers: If "quarterly financial report 2024" fails, try just "financial report"
-   - Try semantic variations: If keyword_search fails, use vector_search which finds conceptually related content
-4. CHAIN TOOLS TOGETHER: Use the results of one tool to inform parameters for the next tool
-5. NEVER respond with "there are no notes about X" until you've tried at least 3 different search variations
-6. DO NOT ask the user what to do next when searches fail - AUTOMATICALLY try different approaches
-7. ALWAYS EXPLAIN what you're doing: "I didn't find results for X, so I'm now searching for Y instead"
-8. If all reasonable search variations fail (minimum 3 attempts), THEN you may inform the user that the information might not be in their notes
+YOU ARE EXPECTED TO USE 10-30 TOOLS PER REQUEST. This is NORMAL and EXPECTED behavior.
+
+TOOL EXECUTION STRATEGY:
+USE BATCH EXECUTION FOR SPEED:
+1. execute_batch([{tool:"search",params:{query:"main topic"}},{tool:"search",params:{query:"related topic"}}])
+2. execute_batch([{tool:"read",params:{noteId:"id1"}},{tool:"read",params:{noteId:"id2"}},{tool:"read",params:{noteId:"id3"}}])
+
+SMART RETRY ON FAILURES:
+- Empty results? → retry_search("original query") automatically tries variations
+- Don't manually retry - use retry_search tool
+
+SIMPLIFIED TOOL NAMES:
+- search (not search_notes) - auto-detects search type
+- read (not read_note) - reads content
+- execute_batch - run multiple tools in parallel
+
+WORKFLOW EXAMPLES:
+A) Comprehensive Search:
+   execute_batch([{tool:"search",params:{query:"AI"}},{tool:"search",params:{query:"machine learning"}},{tool:"search",params:{query:"#important"}}])
+   → execute_batch([{tool:"read",params:{noteId:"..."}} for all found IDs])
+   → retry_search("broader terms") if needed
+
+B) Failed Search Recovery:
+   search("specific term") → empty results 
+   → retry_search("specific term") → auto-tries "term", "concepts", synonyms
+   → execute_batch with all variations
+
+C) Analysis Chain:
+   search → read batch → note_summarization → content_extraction → relationship
+
+ALWAYS USE BATCH EXECUTION when possible - it's much faster than individual tools!
+
+REMEMBER: Users expect THOROUGH exploration. Execute tools rapidly and extensively!
 ```

apps/server/src/assets/llm/prompts/providers/openai_tool_prompt.md

@@ -40,4 +40,32 @@ When responding to queries:
 5. For general questions about the user's notes, provide a summary of all relevant notes found, including brief summaries of individual notes
 6. For specific questions, provide detailed information from the user's notes that directly addresses the question
 7. Always prioritize information from the user's notes over your own knowledge, as the user's notes are likely more up-to-date and personally relevant
+
+CRITICAL: YOU MUST USE 10-30 TOOL CALLS PER REQUEST
+This is expected behavior. Users want comprehensive exploration of their knowledge base.
+
+MANDATORY TOOL USAGE PATTERN:
+1. Use BATCH EXECUTION for parallel processing:
+   <tool_calls>
+   <tool_call id="1"><n>execute_batch</n><parameters>{"tools": [{"tool": "search", "params": {"query": "main topic"}}, {"tool": "search", "params": {"query": "related topic"}}]}</parameters></tool_call>
+   </tool_calls>
+
+2. Read ALL found notes in batches:
+   <tool_calls>
+   <tool_call id="2"><n>execute_batch</n><parameters>{"tools": [{"tool": "read", "params": {"noteId": "id1"}}, {"tool": "read", "params": {"noteId": "id2"}}, {"tool": "read", "params": {"noteId": "id3"}}]}</parameters></tool_call>
+   </tool_calls>
+
+3. Use SMART RETRY for empty results:
+   <tool_calls>
+   <tool_call id="3"><n>retry_search</n><parameters>{"originalQuery": "failed query", "strategy": "all"}</parameters></tool_call>
+   </tool_calls>
+
+SIMPLIFIED TOOL NAMES:
+- search (auto-detects type) instead of search_notes/keyword_search_notes
+- read instead of read_note
+- execute_batch for parallel execution
+- retry_search for automatic variations
+
+WORKFLOW: search batch → read batch → retry if needed → analyze → repeat
+Minimum 10+ tools per request using batch execution for speed!
 ```

apps/server/src/services/llm/pipeline/stages/tool_calling_stage.ts

@@ -483,28 +483,33 @@ export class ToolCallingStage extends BasePipelineStage<ToolExecutionInput, { re
         log.info(`Follow-up needed: ${needsFollowUp}`);
         log.info(`Reasoning: ${hasToolResults ? 'Has tool results to process' : 'No tool results'} ${hasErrors ? ', contains errors' : ''} ${hasEmptyResults ? ', contains empty results' : ''}`);
 
-        // Add a system message with hints for empty results
-        if (hasEmptyResults && needsFollowUp) {
-            log.info('Adding system message requiring the LLM to run additional tools with different parameters');
-
-            // Build a more directive message based on which tools were empty
-            const emptyToolNames = toolResultMessages
-                .filter(msg => this.isEmptyToolResult(msg.content, msg.name || ''))
-                .map(msg => msg.name);
-
-            let directiveMessage = `YOU MUST NOT GIVE UP AFTER A SINGLE EMPTY SEARCH RESULT. `;
-
-            if (emptyToolNames.includes('search_notes') || emptyToolNames.includes('keyword_search')) {
-                directiveMessage += `IMMEDIATELY RUN ANOTHER SEARCH TOOL with broader search terms, alternative keywords, or related concepts. `;
-                directiveMessage += `Try synonyms, more general terms, or related topics. `;
+        // Add aggressive system message for continued tool usage
+        if (needsFollowUp) {
+            log.info('Adding enhanced system message to encourage continued tool usage');
+
+            let directiveMessage = '';
+
+            if (hasEmptyResults) {
+                directiveMessage = `IMPORTANT: No results found with your search. You MUST continue searching with different approaches:
+1. Use discover_tools to find alternative search methods
+2. Try broader search terms or synonyms
+3. Use different search tools (search_notes, keyword_search_notes, attribute_search)
+4. Search for related concepts instead of specific terms
+5. Use read_note on any noteIds you've found previously
+
+CRITICAL: Continue executing tools to find information. Do NOT ask the user for guidance yet - exhaust all search options first.`;
+            } else {
+                directiveMessage = `EXCELLENT! You found ${toolResultMessages.length} results. Now you MUST continue with these actions:
+1. Use read_note with ALL noteId values to get full content
+2. After reading notes, use search_notes or keyword_search_notes to find related information
+3. Use attribute_search to find notes with similar tags/labels
+4. Use note_summarization on long notes
+5. Use content_extraction to pull specific information
+6. Use relationship tool to find connected notes
+
+REMEMBER: Execute multiple tools in sequence to gather comprehensive information. The user expects thorough analysis using 10-20+ tool calls. Continue executing tools!`;
             }
 
-            if (emptyToolNames.includes('keyword_search')) {
-                directiveMessage += `IMMEDIATELY TRY SEARCH_NOTES INSTEAD as it might find matches where keyword search failed. `;
-            }
-
-            directiveMessage += `DO NOT ask the user what to do next or if they want general information. CONTINUE SEARCHING with different parameters.`;
-
             updatedMessages.push({
                 role: 'system',
                 content: directiveMessage
@@ -609,10 +614,13 @@ export class ToolCallingStage extends BasePipelineStage<ToolExecutionInput, { re
             }
         }
 
-        // Add a general suggestion to try search_notes as a fallback
+        // Add general recommendations including new helper tools
         if (!toolName.includes('search_notes')) {
             guidance += "RECOMMENDATION: If specific searches fail, try the 'search_notes' tool which performs semantic searches.\n";
         }
+
+        // Encourage continued tool usage
+        guidance += "\nTry alternative tools immediately. Use discover_tools if unsure which tool to use next.";
 
         return guidance;
     }

apps/server/src/services/llm/tools/attribute_search_tool.ts

@@ -19,26 +19,26 @@ export const attributeSearchToolDefinition: Tool = {
     type: 'function',
     function: {
         name: 'attribute_search',
-        description: 'Search for notes with specific attributes (labels or relations). Use this when you need to find notes based on their metadata rather than content. IMPORTANT: attributeType must be exactly "label" or "relation" (lowercase).',
+        description: 'Search notes by attributes (labels/relations). Finds notes with specific tags, categories, or relationships.',
         parameters: {
             type: 'object',
             properties: {
                 attributeType: {
                     type: 'string',
-                    description: 'MUST be exactly "label" or "relation" (lowercase, no other values are valid)',
+                    description: 'Type of attribute: "label" for tags/categories or "relation" for connections. Case-insensitive.',
                     enum: ['label', 'relation']
                 },
                 attributeName: {
                     type: 'string',
-                    description: 'Name of the attribute to search for (e.g., "important", "todo", "related-to")'
+                    description: 'Name of the attribute (e.g., "important", "todo", "relatedTo").'
                 },
                 attributeValue: {
                     type: 'string',
-                    description: 'Optional value of the attribute. If not provided, will find all notes with the given attribute name.'
+                    description: 'Optional value to match. Leave empty to find all notes with this attribute name.'
                 },
                 maxResults: {
                     type: 'number',
-                    description: 'Maximum number of results to return (default: 20)'
+                    description: 'Maximum number of results (default: 20).'
                 }
             },
             required: ['attributeType', 'attributeName']
@@ -57,13 +57,31 @@ export class AttributeSearchTool implements ToolHandler {
      */
     public async execute(args: { attributeType: string, attributeName: string, attributeValue?: string, maxResults?: number }): Promise<string | object> {
         try {
-            const { attributeType, attributeName, attributeValue, maxResults = 20 } = args;
+            let { attributeType, attributeName, attributeValue, maxResults = 20 } = args;
+
+            // Normalize attributeType to lowercase for case-insensitive handling
+            attributeType = attributeType?.toLowerCase();
 
             log.info(`Executing attribute_search tool - Type: "${attributeType}", Name: "${attributeName}", Value: "${attributeValue || 'any'}", MaxResults: ${maxResults}`);
 
-            // Validate attribute type
+            // Enhanced validation with helpful guidance
             if (attributeType !== 'label' && attributeType !== 'relation') {
-                return `Error: Invalid attribute type. Must be exactly "label" or "relation" (lowercase). You provided: "${attributeType}".`;
+                const suggestions: string[] = [];
+
+                // Check for common variations and provide helpful guidance
+                if (attributeType?.includes('tag') || attributeType?.includes('category')) {
+                    suggestions.push('Use "label" for tags and categories');
+                }
+
+                if (attributeType?.includes('link') || attributeType?.includes('connection')) {
+                    suggestions.push('Use "relation" for links and connections');
+                }
+
+                const errorMessage = `Invalid attributeType: "${attributeType}". Use "label" for tags/categories or "relation" for connections. Examples: 
+- Find tagged notes: {"attributeType": "label", "attributeName": "important"}
+- Find related notes: {"attributeType": "relation", "attributeName": "relatedTo"}`;
+
+                return errorMessage;
             }
 
             // Execute the search