feat!: Add support for real time judge evals #969
Conversation
|
@launchdarkly/browser size report |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@launchdarkly/js-sdk-common size report |
|
@launchdarkly/js-client-sdk size report |
|
@launchdarkly/js-client-sdk-common size report |
|
Will |
If this is set: Then it will only be a minor. |
| })); | ||
| } | ||
| const config = await this._evaluate(key, context, defaultValue, 'completion', variables); | ||
| return this._addVercelAISDKSupport(config as LDAIConversationConfig); |
There was a problem hiding this comment.
It seems like the core AI SDK impl shouldn't know about the specifics of an extending SDK.
There was a problem hiding this comment.
This was previously supported and just left "unchanged" in this PR other than moving it into a helper function. We plan to deprecate this and remove it in support of the new vercel provider package.
There was a problem hiding this comment.
@abarker-launchdarkly do you have any reservations if we remove this now since it will be a major breaking change anyways?
I only want it to be a minor, but I want the proper change logs to show breaking even for the minor bump. |
feat!: AI Config defaults require the "enabled" attribute
feat!: Renamed LDAIAgentConfig to LDAIAgentConfigRequest for improved clarity
feat!: Renamed LDAIAgent to LDAIAgentConfig *note the previous use of this name
feat!: Removed LDAIAgentDefault in favor of LDAIAgentConfig
feat!: Removed LDAIDefaults in favor of LDAIConfig
feat: Added judge method to AI SDK to retrieve an AI Judge Config
feat: Added initJudge method to create a Judge based on the judge key provided
feat: Added trackEvalScores method to config tracker
Note
Introduces Judge evaluations with structured outputs and sampling, adds judgeConfig/createJudge, refactors config into typed modes (completion/agent/judge) with new completionConfig/agentConfig/agentConfigs/createChat, and enhances tracking with eval score metrics.
judgeConfigandcreateJudgefor judge evaluations;ChatResponse.evaluationsreturns async judge results.initChatwithcreateChat; addcompletionConfig,agentConfig, andagentConfigs(deprecateconfig,agent,agents).api/config/types(LDAIConversationConfig*,LDAIAgentConfig*,LDAIJudgeConfig*); remove oldLDAIConfigandagentsmodule.completion/agent/judge) with validation and interpolation; new internalLDAIConfigUtilsfor flag <-> config mapping.TrackedChatnow initializes and runs attached judges in parallel and exposesgetJudges().LDAIConfigTrackeraddsgetTrackData()andtrackEvalScores(); used to emit per-metric judge scores.AIProvideradds defaultinvokeStructuredModelfor judges (and non-abstractinvokeModel); factory updated to accept new config union type.createChatand requireenabledin defaults.Written by Cursor Bugbot for commit 6cc3c43. This will update automatically on new commits. Configure here.