-
-
Notifications
You must be signed in to change notification settings - Fork 616
Open
Description
Is your feature request related to a problem? Please describe.
I'd like to use the metric
property for better filtering of the eval results. My current config looks similar to this one:
promptfooconfig.yaml:
providers:
- file://my_provider.yaml
defaultTest:
assert:
- type: llm-rubric
threshold: 0.1
metric: '{{metric}}'
prompts:
- '{{userPrompt}}'
tests:
- file://my_tests.yaml
my_tests.yaml (contains at least a few hundred prompts):
- vars:
userPrompt: "Some prompt"
metric: "metric1"
- vars:
userPrompt: "Some other prompt"
metric: "other"
Describe the solution you'd like
I'd expect to see the metrics on the UI where I could filter for them, however what I see is just {{metric}}: 50.00% (1.00/2.00)
.
What I'd like to see is: metric1: 50.00% (1.00/2.00) | other: 50.00% (1.00/2.00)
Describe alternatives you've considered
I don't want to expand the assert
for each prompt in the my_tests.yaml
, because I like how it's compact in the defaultTests.assert
Additional context
Promptfoo version: 0.116.7
Metadata
Metadata
Assignees
Labels
No labels