LLM update 202507

[haraldschilly]

LLM Infrastructure Improvements

Key Updates:

1. Google Gemini 2.5 Thinking Token Support - Added maxReasoningTokens: 1024 for Gemini 2.5 models
2. Unified LangChain Implementation - Consolidated 6 LLM provider files into 1 implementation, reducing ~600 lines of duplicated code
3. OpenAI o1 Model Support - Fixed streaming and system role issues for o1 and o1-mini models
4. Enhanced Testing - Added comprehensive user-defined LLM tests and improved admin testing interface
5. Code Consistency & Cleanup - Improved token counting accuracy and error handling

Highlights:

• Fixed OpenAI o1 models - Resolved stream_options and system role compatibility issues
• User-defined LLM tests - Complete test coverage for all supported providers (OpenAI, Google, Anthropic, Mistral, Custom OpenAI)
• Migration framework - Added USE_NEWER_LC_IMPL flag to support gradual migration to unified implementation
• Better testing tools - New table-based test interface with improved error display

All 27 LLM tests now pass, including the new user-defined LLM functionality. The changes maintain backward compatibility while providing a foundation for future LLM improvements.

[williamstein]

@haraldschilly what is the release process? E.g., what's the current coupling between frontend backend for this? You usually do a good job explaining this. E.g., what happens if only the frontend is updated? Only the backend? do I need to force people to refresh?

[haraldschilly]

Hmm, good question. Since the backend reports to the frontend what LLMs are available, it is fine to update the frontend and then the backend. There was no change with the actual communication. Overall, both need to be updated.

[williamstein]

Hmm, good question. Since the backend reports to the frontend what LLMs are available, it is fine to update the frontend and then the backend. There was no change with the actual communication. Overall, both need to be updated.

Thanks - that's optimal ! And of course it is all live right now.