Skip to content

Commit 4aca180

Browse files
committed
Update docs
1 parent 27081dd commit 4aca180

1 file changed

Lines changed: 62 additions & 8 deletions

File tree

docs/evaluate/datasets/evaluations.md

Lines changed: 62 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,74 @@
11
---
22
title: "Running Evaluations"
3-
description: "Fetch hosted datasets and run evaluations with pydantic-evals."
3+
description: "Run evaluations against local or hosted datasets with pydantic-evals."
44
---
55

66
# Running Evaluations
77

8+
Evaluations in Logfire are powered by [pydantic-evals](https://ai.pydantic.dev/evals/). You have two equally supported options for where your test cases live:
9+
10+
- **Local datasets** --- defined in code (or loaded from a YAML file) as a [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. No server round-trip required. This is the simplest way to get started and is all you need for many projects.
11+
- **Hosted datasets** --- stored on Logfire, editable in the [Web UI](ui.md), and fetchable as a typed `Dataset`. Useful when you want to curate cases from production traces or collaborate with teammates.
12+
13+
Either way, once you have a `Dataset` in hand the evaluation step is identical, and results show up in the [Evals](../../guides/web-ui/evals.md) tab as long as Logfire tracing is configured.
14+
815
!!! note "Experimental SDK"
916

10-
The dataset management SDK is under `logfire.experimental.api_client`. The API may change in future releases.
17+
The hosted dataset management SDK is under `logfire.experimental.api_client`. The API may change in future releases. Local datasets use the stable `pydantic-evals` API.
18+
19+
## Evaluating a Local Dataset
20+
21+
If your test cases live in code, you can run an evaluation without ever talking to the Logfire datasets API. Just build a `Dataset` and call `evaluate`:
22+
23+
```python skip-run="true" skip-reason="example-ai-task"
24+
from dataclasses import dataclass
25+
26+
from pydantic_evals import Case, Dataset
27+
28+
29+
@dataclass
30+
class QuestionInput:
31+
question: str
32+
context: str | None = None
33+
34+
35+
@dataclass
36+
class AnswerOutput:
37+
answer: str
38+
confidence: float
39+
40+
41+
dataset = Dataset[QuestionInput, AnswerOutput, None](
42+
cases=[
43+
Case(
44+
name='capital_of_france',
45+
inputs=QuestionInput(question='What is the capital of France?'),
46+
expected_output=AnswerOutput(answer='Paris', confidence=1.0),
47+
),
48+
# ... more cases
49+
],
50+
)
51+
52+
53+
async def my_qa_task(inputs: QuestionInput) -> AnswerOutput:
54+
"""The AI system under test."""
55+
...
56+
57+
58+
async def run_evaluation():
59+
report = await dataset.evaluate(my_qa_task)
60+
report.print()
61+
```
62+
63+
You can also load local datasets from YAML files --- see the [pydantic-evals documentation](https://ai.pydantic.dev/evals/) for details. With Logfire tracing enabled, runs against local datasets still appear in the [Evals](../../guides/web-ui/evals.md) tab (as **Local** datasets --- see [Hosted vs Local Datasets](index.md#hosted-vs-local-datasets)).
64+
65+
## Evaluating a Hosted Dataset
1166

12-
Once you have a hosted dataset (typically published via [`push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted) or created in the [Web UI](ui.md)), you can fetch it as a
13-
typed [`pydantic_evals.Dataset`][pydantic_evals.Dataset] and use it to evaluate your AI system.
67+
If you'd rather manage cases on the server --- for example so teammates can edit them in the UI or so you can seed cases from production traces --- fetch a hosted dataset and use it the same way.
1468

15-
If your dataset only exists locally in code, first publish it with [`client.push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted), then fetch it here to run evaluations against the hosted copy.
69+
Hosted datasets are typically created in the [Web UI](ui.md) or published from code via [`push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted).
1670

17-
## Getting a typed pydantic-evals Dataset
71+
### Getting a typed pydantic-evals Dataset
1872

1973
The `get_dataset` method fetches all hosted cases and returns a typed
2074
[`pydantic_evals.Dataset`][pydantic_evals.Dataset] that you can use directly for evaluation:
@@ -76,9 +130,9 @@ raw_data = client.get_dataset('qa-golden-set')
76130
# raw_data is a dict with 'name', 'cases', etc.
77131
```
78132

79-
## Running the Evaluation
133+
### Running the Evaluation
80134

81-
Use the dataset with pydantic-evals to evaluate your AI system:
135+
Once fetched, a hosted dataset is just a `pydantic_evals.Dataset` --- use it exactly like the local example above:
82136

83137
```python skip="true" skip-reason="external-connection"
84138
from pydantic_evals import Dataset

0 commit comments

Comments
 (0)