Skip to content

AGENTS.md

Latest commit

 

History

History
226 lines (181 loc) · 8.67 KB

AGENTS.md

File metadata and controls

226 lines (181 loc) · 8.67 KB
Raw

AgentEval Development Instructions

This file provides instructions for AI agents working with the AgentEval codebase.

Project Overview

AgentEval is the comprehensive .NET evaluation toolkit for AI agents, built first for Microsoft Agent Framework (MAF). What RAGAS and DeepEval do for Python, AgentEval does for .NET:

  • Tool usage validation with fluent assertions
  • RAG quality metrics (faithfulness, relevance, groundedness)
  • stochastic evaluation and statistical model comparison
  • Behavioral policies (NeverCallTool, MustConfirmBefore)
  • Trace record/replay for deterministic CI testing

Quick Reference

Build & Test

dotnet build                    # Build all projects
dotnet test                     # Run all tests (×3 TFMs)
dotnet run --project samples/AgentEval.Samples

Key Directories

  • src/AgentEval/ - Umbrella packaging project (NuGet: AgentEval)
  • src/AgentEval.Abstractions/ - Public contracts: interfaces, models
  • src/AgentEval.Core/ - Implementations: metrics, assertions, comparison, tracing
  • src/AgentEval.DataLoaders/ - Data loaders, exporters, output formatting
  • src/AgentEval.MAF/ - Microsoft Agent Framework integration
  • src/AgentEval.Memory/ - Memory evaluation (retention, temporal, cross-session, benchmarks)
  • src/AgentEval.RedTeam/ - Security scanning, attack types, compliance
  • tests/AgentEval.Tests/ - Unit tests (mirrors src structure)
  • tests/AgentEval.Memory.Tests/ - Memory module unit tests
  • samples/AgentEval.Samples/ - 41 runnable samples
  • docs/ - Documentation

Core Interfaces

  • IMetric → IRAGMetric, IAgenticMetric
  • IEvaluableAgent → IStreamableAgent
  • IEvaluationHarness → MAFEvaluationHarness

Metric Naming

  • llm_* = LLM-evaluated (API cost)
  • code_* = Code-computed (free)
  • embed_* = Embedding-based

Assertions Entry Points

result.ToolUsage!.Should()    // Tool assertions
result.Performance!.Should()   // Performance assertions
result.ActualOutput!.Should()  // Response assertions

Code Conventions

C# Style

  • C# preview features enabled (LangVersion=preview)
  • Nullable enabled throughout
  • File-scoped namespaces preferred
  • required properties for models
  • XML docs on public APIs

Test Naming

MethodName_StateUnderTest_ExpectedBehavior

Error Messages

Must include: Expected, Actual, Suggestions, Because parameter

Architectural Principles

This codebase follows SOLID, DRY, KISS, CLEAN principles strictly. See docs/adr/006-service-based-architecture-di.md for details.

Key Points

  • Interface-First: All core services have interfaces (IStochasticRunner, IModelComparer, etc.)
  • DI/IOC: Use services.AddAgentEval() for registration; inject interfaces, not implementations
  • No Over-Engineering: Don't add interfaces for builders, config objects, or test-time tools
  • Single Responsibility: Each service has one focused purpose

DI Pattern

// Register all services (umbrella convenience):
services.AddAgentEvalAll();

// Or register selectively:
services.AddAgentEval();              // Core services
services.AddAgentEvalDataLoaders();   // DataLoaders + Exporters
services.AddAgentEvalMemory();        // Memory evaluation
services.AddAgentEvalRedTeam();       // Red Team security testing

// Inject interfaces
public class MyService(IStochasticRunner runner, IModelComparer comparer) { }

Common Tasks

Adding a New Metric

  1. Create in appropriate sub-project:
    • src/AgentEval.Core/Metrics/RAG/ for RAG metrics
    • src/AgentEval.Core/Metrics/Agentic/ for agentic metrics
  2. Implement IRAGMetric or IAgenticMetric
  3. Use appropriate prefix (llm_/code_/embed_)
  4. Add tests using FakeChatClient in tests/AgentEval.Tests/Metrics/

Adding a New Assertion

  1. Add to ToolUsageAssertions, PerformanceAssertions, or ResponseAssertions
  2. Use [StackTraceHidden] attribute
  3. Call AgentEvalScope.FailWith() with structured exception

Adding a New Sample

  1. Create SampleXX_Name.cs in samples project
  2. Register in Program.cs menu
  3. Follow header template with time estimate
  4. Provide mock fallback for offline testing

Testing Without LLM

Use FakeChatClient for unit tests:

var fakeClient = new FakeChatClient("""{"score": 95}""");
var metric = new FaithfulnessMetric(fakeClient);

Use Trace Replay for integration tests:

var trace = await TraceSerializer.LoadFromFileAsync("saved-trace.json");
var replayer = new TraceReplayingAgent(trace);
var response = await replayer.InvokeAsync("prompt");

Git Commit Conventions

Skip CI for Intermediate Commits

When making multiple small changes (docs, formatting, etc.), use [skip ci] to avoid triggering full CI pipeline:

git commit -m "docs: Fix typo in README [skip ci]"
git commit -m "style: Format code [skip ci]"

When to use [skip ci]:

  • Documentation-only changes
  • Comment/typo fixes
  • Formatting changes
  • Intermediate commits during a multi-commit refactor

When NOT to use [skip ci]:

  • Any code changes (even small ones)
  • Final commit of a feature/fix
  • Changes to tests, samples, or configuration

Commit Message Format

Use conventional commits:

  • feat: New feature
  • fix: Bug fix
  • docs: Documentation only
  • style: Formatting, no code change
  • refactor: Code restructuring
  • test: Adding/updating tests
  • chore: Maintenance tasks

Environment Variables

For samples/integration tests:

AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_DEPLOYMENT=gpt-4o

Microsoft Agent Framework — MAF Doctor is installed

This repository uses Microsoft Agent Framework. The maf-doctor MCP server (also called "MAF Doctor") is installed and exposes tools for diagnosing, fixing, and migrating MAF code.

Before answering MAF questions or proposing changes:

  1. Always call MafDoctor first on the repo path to get the current health grade (A-F) and the top issues. Don't speculate about MAF quality without this baseline.

  2. For any [Obsolete] warning, CS0618 / CS0246 diagnostic, or build failure mentioning a MAF type — call MafRunCs0618Hunt (full project scan) or MafApiSafety (single symbol) BEFORE suggesting a fix. The maf-doctor registry has curated fix recipes that supersede your training data — MAF ships breaking changes every minor version, so training data is likely outdated.

  3. To fix issues — MafAutoFixAll --dry-run then apply handles the mechanical rules deterministically (offer this first; the rewrites are tested). To fix everything, run the maf-remediate prompt (or just ask "fix all the issues maf-doctor found"): it grades → plans → autofixes → then works each semantic finding. Every finding carries a confidence (certain / high / heuristic); a heuristic finding may be a false positive — confirm it with MafExplainFinding before editing. Get the plan via MafDoctor(format: "plan") (human) or --plan --json (structured manifest); per-rule fix + false-positive guidance lives in the maf-remediation-playbook skill.

  4. When designing a new MAF agent or workflow — call MafNewAgent / MafNewExecutor for scaffolds, or MafSimulateWorkflow for topology preview. Don't reconstruct patterns from memory.

  5. For deep architectural / security / migration questions — run the maf-review (best-practice audit), maf-audit (pre-migration scan + plan), maf-migrate (execute a migration-plan.md), or maf-debug (diagnose a symptom) MCP prompts — init wires these up automatically, no extra setup needed. If .github/agents/ was also copied in from the maf-doctor source repo (GitHub Copilot only — see the toolkit's init-reference.md), the equivalent @maf-best-practice-reviewer, @maf-auditor, @maf-migration, or @maf-incident-responder personas work the same way.

  6. To migrate FROM Semantic Kernel TO MAF (a cross-framework port, NOT a MAF version bump) — call MafDetectSourceFramework (CLI: maf-doctor migrate-scan) to inventory SK usage and scope it, then run the maf-migrate-from prompt (or the @maf-cross-migration agent, if .github/agents/ was also copied in). It scaffolds a new MAF project beside the original and ports it construct-by-construct, non-destructively. The mapping lives at maf://migrate-from?source=semantic-kernel.

maf-doctor tools are MAF-version-aware via applies_to_codebases markers in the registry — they know which fix applies to which MAF version. Defer to the tools.