Fix issue #2044: Improve output parsing and timeout for local LLMs #2046
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit addresses the issues reported in #2044 where local LLMs like Ollama with qwen2.5:14b were encountering RagasOutputParserException and TimeoutError during evaluation. Changes: 1. Increased default timeout in RunConfig from 180 to 300 seconds to accommodate slower local LLMs 2. Enhanced extract_json function to better handle various LLM output formats: - Added support for markdown code blocks - Added handling for single quotes in JSON - Added handling for trailing commas in JSON - Improved JSON extraction from text with surrounding content 3. Improved RagasOutputParser with: - More robust error handling - Increased default retries from 1 to 3 - Better error messages with details about the failure - Added suggestions for working with local LLMs in error messages Added tests: - Tests for extract_json with various input formats - Tests for RagasOutputParser fallback mechanism - Tests for timeout configuration
dosubot
bot
added
the
size:L
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the issues reported in #2044 where local LLMs like Ollama with qwen2.5:14b were encountering RagasOutputParserException and TimeoutError during evaluation.
Changes
Added tests
Fixes #2044
@jjmachan can click here to continue refining the PR