feat: add support for job attachment evaluation

[AAgnihotry]

Summary

This PR adds support for evaluating agent outputs stored as job attachments. When an agent returns a job attachment URI, evaluators can now automatically download and evaluate the attachment content.

Key Features

• Automatic attachment detection: Evaluators detect URIs matching urn:uipath:cas:file:orchestrator:{uuid}
• Seamless download: Automatically downloads attachment content using AttachmentsService
• Universal support: Works with all output evaluators (ExactMatch, Contains, LineByLine, etc.)
• Comprehensive testing: 23 new tests with 100% coverage
• Working sample: Complete end-to-end example demonstrating the feature

Changes

Core Implementation

1. src/uipath/eval/evaluators/attachment_utils.py (new)

   • is_job_attachment_uri() - Detects attachment URI pattern
   • extract_attachment_id() - Extracts UUID from URI
   • download_attachment_as_string() - Downloads and returns content

2. src/uipath/eval/evaluators/output_evaluator.py

   • Updated _get_actual_output() to detect and download attachments

3. src/uipath/eval/evaluators/base_legacy_evaluator.py

   • Updated _get_actual_output() with attachment support

4. tests/evaluators/test_attachment_utils.py (new)

   • 23 comprehensive tests covering all utility functions
   • Tests for URI validation, extraction, download, and error handling

Sample Project

Complete working sample in samples/attachment_evaluation_test/:

• Agent (main.py): Generates reports and uploads as attachments
• Evaluators: 3 configurations (ExactMatch, Contains, LineByLine)
• Test Cases: 5 test scenarios with expected outputs
• Documentation: README.md and QUICKSTART.md

How It Works

# Agent returns attachment URI
return Output(
    report="urn:uipath:cas:file:orchestrator:123e4567-...",
    status="completed"
)

# Evaluator automatically:
# 1. Detects the URI pattern
# 2. Extracts the attachment ID
# 3. Downloads the content
# 4. Evaluates against expected criteria

Test Results

All 1678 tests pass, including 23 new attachment utility tests:

✓ URI detection (valid/invalid patterns)
✓ UUID extraction
✓ Attachment download (success/failure)
✓ Temporary file cleanup
✓ Error handling

Sample Evaluation Results

┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃  Evaluation       ┃  ReportContentM…  ┃  ReportContain…  ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│  Test sales       │              1.0  │             1.0  │
│  report exact     │                   │                  │
│  match            │                   │                  │
│  Test inventory   │                -  │             1.0  │
│  report contains  │                   │                  │
│  Test generic     │              1.0  │             1.0  │
│  report exact     │                   │                  │
│  match            │                   │                  │
└───────────────────┴───────────────────┴──────────────────┘

Version

• Bumped to 2.10.31

Testing

To test locally:

cd samples/attachment_evaluation_test
export UIPATH_URL="https://your-tenant.uipath.com"
export UIPATH_ACCESS_TOKEN="your-token"
uipath eval main evaluations/eval-sets/default.json --workers 1

🤖 Generated with Claude Code

Development Packages

uipath

[project]
dependencies = [
  # Exact version:
  "uipath==2.10.33.dev1014895667",

  # Any version from PR
  "uipath>=2.10.33.dev1014890000,<2.10.33.dev1014900000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[Chibionos]

hide with a feature flag and make sure this is isolated.

[Chibionos]

hide with a feature flag and make sure this is isolated.

[AAgnihotry]