feat(llm): add configurable LLMCustomRule evaluator#397
feat(llm): add configurable LLMCustomRule evaluator#397e06084 merged 13 commits intoMigoXLab:devfrom
Conversation
feat: v2.2.2
…lated runtime config
There was a problem hiding this comment.
Code Review
This pull request introduces the LLMCustomRule evaluator, which enables users to define custom LLM-based quality metrics via configuration. The changes include new Pydantic schemas for custom rules, logic for prompt construction and response parsing, and updates to the model configuration system to ensure deep copying and isolation of dynamic settings. Review feedback identifies a non-deterministic fallback when the model name is missing, a bug in the default exception name initialization, and an opportunity to simplify configuration dictionary creation using Pydantic's native methods.
| if self.dynamic_config.model: | ||
| model_name = self.dynamic_config.model | ||
| else: | ||
| model_name = self.client.models.list().data[0].id |
There was a problem hiding this comment.
Fetching the model list and picking the first available ID as a fallback is non-deterministic and inefficient. The first model returned by the API might not support chat completions (e.g., it could be an embedding or image model), which would cause a runtime error. Additionally, this adds an extra network request for every evaluation if the model is not specified in the config. It is recommended to either require the model field in the configuration or provide a sensible hardcoded default (e.g., gpt-4o-mini).
|
|
||
| attempts = 0 | ||
| except_msg = "" | ||
| except_name = Exception.__class__.__name__ |
There was a problem hiding this comment.
This initializes except_name to 'type' because Exception.__class__ refers to the type class. While this variable is likely to be overwritten in the exception handlers, it is better to use Exception.__name__ to correctly default to the string 'Exception' for clarity.
| except_name = Exception.__class__.__name__ | |
| except_name = Exception.__name__ |
| config_items = { | ||
| field_name: getattr(llm_config, field_name) | ||
| for field_name in llm_config.__class__.model_fields | ||
| } | ||
| config_items.update(llm_config.model_extra) |
There was a problem hiding this comment.
This logic can be simplified using dict(llm_config). In Pydantic v2, calling dict() on a model instance returns a shallow dictionary containing both the defined fields and any extra attributes (if extra='allow' is set), which achieves the goal of preserving nested Pydantic objects while including extra fields.
config_items = dict(llm_config)
Summary
Validation