Skip to content

Reasoning support for evaluators #42482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 108 commits into from
Closed

Reasoning support for evaluators #42482

wants to merge 108 commits into from

Conversation

@nagkumar91
Copy link
Member

@nagkumar91 nagkumar91 commented Aug 12, 2025 •
edited
Loading

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Nagkumar Arkalgud and others added 30 commits May 28, 2025 11:11
Prepare evals SDK Release
4318329
Fix bug
192b980
Fix for ADV_CONV for FDP projects
758adb4
Update release date
de09fd1
@nagkumar91
Merge branch 'main' into main
ef60fe6
@nagkumar91
Merge branch 'Azure:main' into main
8ca51d0
@nagkumar91
Merge branch 'Azure:main' into main
98bfc3a
@nagkumar91
Merge branch 'Azure:main' into main
a5f32e8
@nagkumar91
Merge branch 'Azure:main' into main
5fd88b6
@nagkumar91
Merge branch 'Azure:main' into main
51f2b44
@nagkumar91
Merge branch 'Azure:main' into main
a5be8b5
@nagkumar91
Merge branch 'Azure:main' into main
75965b7
@nagkumar91
Merge branch 'Azure:main' into main
d0c5e53
@nagkumar91
Merge branch 'Azure:main' into main
b790276
@nagkumar91
Merge branch 'Azure:main' into main
d5ca243
re-add pyrit to matrix
8d62e36
Change grader ids
59a70f2
@nagkumar91
Merge branch 'Azure:main' into main
4d146d7
Update unit test
f7a4c83
replace all old grader IDs in tests
79e3a40
@nagkumar91
Merge branch 'main' into main
588cbec
@nagkumar91
Update platform-matrix.json
7514472 
Add pyrit and not remove the other one
Update test to ensure everything is mocked
28b2513
tox/black fixes
8603e0e
Skip that test with issues
895f226
@nagkumar91
Merge branch 'Azure:main' into main
b4b2daf
update grader ID according to API View feedback
023f07f
Update test
45b5f5d
remove string check for grader ID
1ccb4db
@nagkumar91
Merge branch 'Azure:main' into main
6fd9aa5
@nagkumar91 nagkumar91 requested a review from Copilot September 24, 2025 15:15
Copilot AI reviewed Sep 24, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

@nagkumar91 nagkumar91 requested a review from Copilot September 25, 2025 15:09
Copilot AI reviewed Sep 25, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Nagkumar Arkalgud added 3 commits September 25, 2025 08:47
Evaluation refactors and cleanup:
2f43619 
- Simplify client selection: default code_client; support _use_pf_client/_use_run_submitter_client with conflict check.
- Groundedness: always pass is_reasoning_model to AsyncPrompty when switching templates.
- Remove stray debug prints in CodeClient.get_metrics.
- Tidy imports in groundedness evaluator (separate os/logging).
- Reasoning model params: robust dict handling for parameters in AsyncPrompty to avoid dict() pitfalls.
Apply evaluation refactors and cleanup into diff branch
40fb39c
Revert client selection to legacy tri-state behavior to satisfy tests…
2f2ceec 
…; keep other refactors intact.
@nagkumar91 nagkumar91 requested a review from Copilot September 25, 2025 19:03
Copilot AI reviewed Sep 25, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

@nagkumar91 nagkumar91 enabled auto-merge (squash) September 30, 2025 14:01
@nagkumar91 nagkumar91 disabled auto-merge September 30, 2025 14:34
@nagkumar91 nagkumar91 closed this Oct 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@luigiw luigiw luigiw approved these changes

@slister1001 slister1001 slister1001 approved these changes

@singankit singankit Awaiting requested review from singankit

Assignees

No one assigned

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nagkumar91 @luigiw @singankit @slister1001