Skip to content

Fix task config metric typing to accept Metric enums #1018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+2 −2
Open

Fix task config metric typing to accept Metric enums #1018

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd9f8c2
Fix task config metric typing to accept Metric enums
emmanuel-ferdman Oct 14, 2025
5b94ef0
Merge branch 'main' into main
NathanHB Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/lighteval/tasks/lighteval_task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ class LightevalTaskConfig:
name as input.
hf_repo (str): HuggingFace Hub repository path containing the evaluation dataset.
hf_subset (str): Dataset subset/configuration name to use for this task.
metrics (ListLike[Metric]): List of metrics to compute for this task.
metrics (ListLike[Metric | Metrics]): List of metrics or metric enums to compute for this task.

Dataset Configuration:
hf_revision (str | None, optional): Specific dataset revision to use.
Expand Down Expand Up @@ -112,7 +112,7 @@ class LightevalTaskConfig:
] # The prompt function should be used to map a line in the dataset to a Sample
hf_repo: str
hf_subset: str
metrics: ListLike[Metric] # List of metric , should be configurable
metrics: ListLike[Metric | Metrics] # Accept both Metric objects and Metrics enums

# Additional hf dataset config
hf_revision: str | None = None
Expand Down
Loading