Skip to content

Use {{metric}} variable in defaultTests.assert #4986

@vira-s

Description

@vira-s

Is your feature request related to a problem? Please describe.
I'd like to use the metric property for better filtering of the eval results. My current config looks similar to this one:
promptfooconfig.yaml:

providers:
  - file://my_provider.yaml
defaultTest:
  assert:
    - type: llm-rubric
      threshold: 0.1
      metric: '{{metric}}'
prompts:
  - '{{userPrompt}}'
tests:
  - file://my_tests.yaml

my_tests.yaml (contains at least a few hundred prompts):

- vars:
  userPrompt: "Some prompt"
  metric: "metric1"
- vars:
  userPrompt: "Some other prompt"
  metric: "other"

Describe the solution you'd like
I'd expect to see the metrics on the UI where I could filter for them, however what I see is just {{metric}}: 50.00% (1.00/2.00).
What I'd like to see is: metric1: 50.00% (1.00/2.00) | other: 50.00% (1.00/2.00)

Describe alternatives you've considered
I don't want to expand the assert for each prompt in the my_tests.yaml, because I like how it's compact in the defaultTests.assert

Additional context
Promptfoo version: 0.116.7

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions