|
1 | 1 | --- |
2 | 2 | title: "Running Evaluations" |
3 | | -description: "Fetch hosted datasets and run evaluations with pydantic-evals." |
| 3 | +description: "Run evaluations against local or hosted datasets with pydantic-evals." |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | # Running Evaluations |
7 | 7 |
|
| 8 | +Evaluations in Logfire are powered by [pydantic-evals](https://ai.pydantic.dev/evals/). You have two equally supported options for where your test cases live: |
| 9 | + |
| 10 | +- **Local datasets** --- defined in code (or loaded from a YAML file) as a [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. No server round-trip required. This is the simplest way to get started and is all you need for many projects. |
| 11 | +- **Hosted datasets** --- stored on Logfire, editable in the [Web UI](ui.md), and fetchable as a typed `Dataset`. Useful when you want to curate cases from production traces or collaborate with teammates. |
| 12 | + |
| 13 | +Either way, once you have a `Dataset` in hand the evaluation step is identical, and results show up in the [Evals](../../guides/web-ui/evals.md) tab as long as Logfire tracing is configured. |
| 14 | + |
8 | 15 | !!! note "Experimental SDK" |
9 | 16 |
|
10 | | - The dataset management SDK is under `logfire.experimental.api_client`. The API may change in future releases. |
| 17 | + The hosted dataset management SDK is under `logfire.experimental.api_client`. The API may change in future releases. Local datasets use the stable `pydantic-evals` API. |
| 18 | + |
| 19 | +## Evaluating a Local Dataset |
| 20 | + |
| 21 | +If your test cases live in code, you can run an evaluation without ever talking to the Logfire datasets API. Just build a `Dataset` and call `evaluate`: |
| 22 | + |
| 23 | +```python skip-run="true" skip-reason="example-ai-task" |
| 24 | +from dataclasses import dataclass |
| 25 | + |
| 26 | +from pydantic_evals import Case, Dataset |
| 27 | + |
| 28 | + |
| 29 | +@dataclass |
| 30 | +class QuestionInput: |
| 31 | + question: str |
| 32 | + context: str | None = None |
| 33 | + |
| 34 | + |
| 35 | +@dataclass |
| 36 | +class AnswerOutput: |
| 37 | + answer: str |
| 38 | + confidence: float |
| 39 | + |
| 40 | + |
| 41 | +dataset = Dataset[QuestionInput, AnswerOutput, None]( |
| 42 | + cases=[ |
| 43 | + Case( |
| 44 | + name='capital_of_france', |
| 45 | + inputs=QuestionInput(question='What is the capital of France?'), |
| 46 | + expected_output=AnswerOutput(answer='Paris', confidence=1.0), |
| 47 | + ), |
| 48 | + # ... more cases |
| 49 | + ], |
| 50 | +) |
| 51 | + |
| 52 | + |
| 53 | +async def my_qa_task(inputs: QuestionInput) -> AnswerOutput: |
| 54 | + """The AI system under test.""" |
| 55 | + ... |
| 56 | + |
| 57 | + |
| 58 | +async def run_evaluation(): |
| 59 | + report = await dataset.evaluate(my_qa_task) |
| 60 | + report.print() |
| 61 | +``` |
| 62 | + |
| 63 | +You can also load local datasets from YAML files --- see the [pydantic-evals documentation](https://ai.pydantic.dev/evals/) for details. With Logfire tracing enabled, runs against local datasets still appear in the [Evals](../../guides/web-ui/evals.md) tab (as **Local** datasets --- see [Hosted vs Local Datasets](index.md#hosted-vs-local-datasets)). |
| 64 | + |
| 65 | +## Evaluating a Hosted Dataset |
11 | 66 |
|
12 | | -Once you have a hosted dataset (typically published via [`push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted) or created in the [Web UI](ui.md)), you can fetch it as a |
13 | | -typed [`pydantic_evals.Dataset`][pydantic_evals.Dataset] and use it to evaluate your AI system. |
| 67 | +If you'd rather manage cases on the server --- for example so teammates can edit them in the UI or so you can seed cases from production traces --- fetch a hosted dataset and use it the same way. |
14 | 68 |
|
15 | | -If your dataset only exists locally in code, first publish it with [`client.push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted), then fetch it here to run evaluations against the hosted copy. |
| 69 | +Hosted datasets are typically created in the [Web UI](ui.md) or published from code via [`push_dataset(...)`](sdk.md#publishing-a-local-typed-dataset-to-hosted). |
16 | 70 |
|
17 | | -## Getting a typed pydantic-evals Dataset |
| 71 | +### Getting a typed pydantic-evals Dataset |
18 | 72 |
|
19 | 73 | The `get_dataset` method fetches all hosted cases and returns a typed |
20 | 74 | [`pydantic_evals.Dataset`][pydantic_evals.Dataset] that you can use directly for evaluation: |
@@ -76,9 +130,9 @@ raw_data = client.get_dataset('qa-golden-set') |
76 | 130 | # raw_data is a dict with 'name', 'cases', etc. |
77 | 131 | ``` |
78 | 132 |
|
79 | | -## Running the Evaluation |
| 133 | +### Running the Evaluation |
80 | 134 |
|
81 | | -Use the dataset with pydantic-evals to evaluate your AI system: |
| 135 | +Once fetched, a hosted dataset is just a `pydantic_evals.Dataset` --- use it exactly like the local example above: |
82 | 136 |
|
83 | 137 | ```python skip="true" skip-reason="external-connection" |
84 | 138 | from pydantic_evals import Dataset |
|
0 commit comments