Skip to content

Conversation

m7mdhka
Copy link
Contributor

@m7mdhka m7mdhka commented Sep 29, 2025

Description:

This PR enhances the evaluate_sync method in the Dataset class to accept retry_task and retry_evaluators arguments. Previously, these arguments could only be passed to the asynchronous evaluate method, limiting the ability to configure retry behavior in synchronous workflows.

This change forwards the retry configurations to the underlying evaluate method, allowing for more robust and convenient error handling in synchronous evaluations.

Changes include:

  • Added retry_task and retry_evaluators parameters to evaluate_sync.
  • Passed these parameters to the evaluate method.
  • Included a new test case to verify the functionality.

Closes #3010

@DouweM DouweM changed the title Feature: forward retry args to evaluate_sync Add retry args to pydantic_evals.Dataset.evaluate_sync Sep 29, 2025
@DouweM
Copy link
Collaborator

DouweM commented Sep 29, 2025

@m7mdhka Please have a look at the failing tests. I don't think we really need them, for what it's worth, as the actual functionality is already tested on the evaluate method.

@m7mdhka m7mdhka force-pushed the feat/forward-retry-config-to-evaluate-sync branch from bbd5d62 to ea723c7 Compare September 30, 2025 08:17
@m7mdhka
Copy link
Contributor Author

m7mdhka commented Sep 30, 2025

@DouweM Yes, you're correct. I removed the test functionality. Everything is good now!

@DouweM DouweM merged commit fd6e86e into pydantic:main Sep 30, 2025
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

evaluate_sync method on Dataset should support retry_task and retry_evaluators arguments
2 participants