Skip to content

Commit b1f2461

Browse files
klaidermanklaiderman
committed
feat(recipes): cross-runtime parity (LangChain, Pydantic-AI, BAML) + eval-harnesses + index
- langchain: middleware list enforcement, LangGraph interrupts, three propagation patterns - pydantic-ai: Agent[Deps,Output] generics, output validation, tool retries (no clone() inheritance documented) - baml: function-shaped LLM calls, Jinja prompt blocks, BamlError - paradigm note acknowledges no separate system-prompt slot - eval-harnesses: 9-suite reference table mapping suites to framework modules; selection guide - recipes/README: index for the 8 recipes (claude-code, openai-agents, cursor, langchain, pydantic-ai, baml, system-prompt, eval-harnesses)
1 parent 90e39c0 commit b1f2461

5 files changed

Lines changed: 774 additions & 0 deletions

File tree

‎recipes/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Recipes
2+
3+
Adoption guides for host platforms. Each recipe gives concrete file paths, concrete config, and verification steps for wiring agent-foundations into a specific host. Unlike the conduct modules (which say *what* to do) and the engines (which say *what to compute*), recipes say *where to put it* — per-host specifics that would be wrong to bake into the portable layers.
4+
5+
## Index
6+
7+
| Recipe | What it covers |
8+
|--------|---------------|
9+
| [claude-code.md](claude-code.md) | Wiring conduct modules and hooks into Claude Code via `CLAUDE.md` and `.claude/settings.json` |
10+
| [openai-agents.md](openai-agents.md) | Integrating the framework with the OpenAI Agents SDK: tool descriptors, handoffs, and guardrails |
11+
| [cursor.md](cursor.md) | Dropping conduct modules into Cursor via `.cursor/rules/` and `.mdc` frontmatter |
12+
| [langchain.md](langchain.md) | Adopting the framework inside a LangChain agent pipeline: chain structure and callback hooks |
13+
| [pydantic-ai.md](pydantic-ai.md) | Wiring conduct and engines into a Pydantic AI agent: validators, deps, and result types |
14+
| [baml.md](baml.md) | Using BAML for structured output extraction in agent pipelines: schema authoring and BamlError handling |
15+
| [system-prompt.md](system-prompt.md) | Loading conduct modules via a system prompt when no framework-native integration is available |
16+
| [eval-harnesses.md](eval-harnesses.md) | Connecting agent-foundations to an eval harness: failure-code tagging, axis aggregation, and regression gates |
17+
18+
## How to read
19+
20+
1. **Match your host** — find the recipe for the platform or SDK you're adopting into.
21+
2. **Read the recipe end-to-end** — each recipe is self-contained; it names the files to create, the config to set, and the check to run to confirm the integration is live.
22+
3. **Follow cross-references** — recipes link into `conduct/`, `engines/`, and `taxonomy/` for the rules and math that back the wiring. Read those docs for the *why*; the recipe gives the *how*.

‎recipes/baml.md‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Recipe — BAML
2+
3+
Audience: teams using BAML (Boundary ML) for typed LLM calls. **BAML's paradigm differs fundamentally from agent SDKs**: it does not expose a multi-turn agent loop or a separate system-prompt slot. LLM calls are shaped as typed functions, prompts live inside Jinja template blocks, and structured output is enforced by the BAML parser. Conduct modules adopt by being included in the prompt block of relevant functions — not by being injected as a separate system message.
4+
5+
Read the paradigm note below before attempting to wire conduct the way you would in an OpenAI or LangChain recipe. The adoption pattern is different by design.
6+
7+
## Wire it up
8+
9+
```bash
10+
git submodule add https://github.com/enchanter-ai/agent-foundations vendor/foundations
11+
```
12+
13+
Define a BAML function that includes conduct modules via Jinja:
14+
15+
```baml
16+
// vendor/foundations/conduct modules are plain Markdown — include them as Jinja partials.
17+
// Render via your project's Jinja environment before passing to the BAML compiler,
18+
// or use a shared partial loader that maps conduct module names to their file contents.
19+
20+
// "function name(parameters) -> return_type {
21+
// client llm_specification
22+
// prompt block_string_specification
23+
// }"
24+
// — https://docs.boundaryml.com/ref/baml/function.mdx
25+
//
26+
// "Input parameters with explicit types, a return type specification, an LLM client, a prompt"
27+
// — https://docs.boundaryml.com/ref/baml/function.mdx
28+
29+
function ImproveCode(code: string, task: string) -> CodeReview {
30+
client GPT4o
31+
prompt #"
32+
{% include 'conduct/discipline.md' %}
33+
{% include 'conduct/verification.md' %}
34+
35+
## Task
36+
Review the following code with respect to the conduct rules above.
37+
Task description: {{ task }}
38+
39+
## Code
40+
{{ code }}
41+
42+
Return a CodeReview with fields: summary, issues (list), verdict.
43+
"#
44+
}
45+
46+
class CodeReview {
47+
summary string
48+
issues string[]
49+
verdict "approve" | "hold" | "reject"
50+
}
51+
```
52+
53+
The conduct text is part of the prompt template. Every call to `ImproveCode` pays the token cost of the included modules for that call.
54+
55+
## Paradigm note
56+
57+
BAML's adoption pattern is shaped by the framework's design. Before wiring conduct, understand these three differences from agent-SDK recipes:
58+
59+
**No multi-turn agent.** Each BAML function is a single LLM call. Conduct rules apply per-call; multi-turn protections (sycophancy resistance, doubt-engine, context checkpointing) are the orchestrator's responsibility — the Python, TypeScript, or Ruby caller that invokes the BAML function, not BAML itself.
60+
61+
**No separate instructions slot.** The conduct text is part of the Jinja prompt template. There is no `system` vs. `user` split analogous to the OpenAI chat API or Pydantic-AI's `instructions` field. This means:
62+
63+
- Token cost is paid on every function call, not amortized across turns of an agent session.
64+
- Pull in only the modules the function needs — full-inherit is expensive per call.
65+
- Module placement within the prompt template matters (see [`../conduct/context.md`](../conduct/context.md) § U-curve placement). Put load-bearing rules near the top or bottom of the prompt block.
66+
67+
**Strong typing as the primary enforcement mechanism.** The return type in the function signature plus the BAML parser's retry loop is the core enforcement mechanism — analogous to Pydantic-AI's output validation but operating at the `.baml` source level before any Python code runs.
68+
69+
## Tier mapping
70+
71+
BAML's `client` block declares the LLM. Map the framework's three tiers to whichever providers your project uses:
72+
73+
| Tier | Role | Example client declarations |
74+
|------|------|-----------------------------|
75+
| Top-tier | Orchestration, judgment, technique selection | `client ClaudeOpus { provider anthropic model claude-opus-4-7 }` |
76+
| Mid-tier | Convergence loops, adversarial passes, translation | `client ClaudeSonnet { provider anthropic model claude-sonnet-4-6 }`, `client GPT4o { provider openai model gpt-4o }` |
77+
| Low-tier | Shape checks, extraction, fetch, freshness audits | `client ClaudeHaiku { provider anthropic model claude-haiku-3 }`, `client GPT4oMini { provider openai model gpt-4o-mini }` |
78+
79+
Calibrate prompt density per [`../conduct/tier-sizing.md`](../conduct/tier-sizing.md) — low-tier functions need mechanical steps in the prompt block; top-tier functions run on intent.
80+
81+
## Enforcement wiring
82+
83+
**Output type as the gate.** Define a typed return in the function signature; BAML's parser enforces the schema on every response:
84+
85+
> "Raised when BAML fails to parse a string from the LLM into the specified object."
86+
> — https://docs.boundaryml.com/guide/baml-basics/error-handling.mdx
87+
88+
> "Our parser is very forgiving, allowing for structured data parsing even in the presence of minor errors."
89+
> — https://docs.boundaryml.com/guide/baml-basics/error-handling.mdx
90+
91+
> "When BAML raises an exception, it will be an instance of a subclass of `BamlError`."
92+
> — https://docs.boundaryml.com/guide/baml-basics/error-handling.mdx
93+
94+
Handle parse failures explicitly in your calling code:
95+
96+
```python
97+
from baml_client import b
98+
from baml_client.types import BamlError, BamlValidationError
99+
100+
try:
101+
review = b.ImproveCode(code=source, task=task_description)
102+
except BamlValidationError as e:
103+
# Parse failure → F02 Fabrication (model emitted non-conforming output)
104+
log_failure(code="F02", evidence=str(e), counter="tighten the return type or add examples")
105+
raise
106+
except BamlError as e:
107+
# Other BAML-layer error (client failure, timeout, etc.)
108+
raise
109+
```
110+
111+
Map `BamlValidationError` to `failure-modes.md` code F02 (Fabrication) — the model emitted output that does not conform to the declared return type.
112+
113+
**Honest limit.** The BAML parser is described as "very forgiving, allowing for structured data parsing even in the presence of minor errors." For load-bearing or destructive operations, add a strict re-validation step in your calling code after BAML parsing succeeds — the parser's leniency is a feature for high-recall extraction, not a substitute for strict validation in high-stakes paths.
114+
115+
**Conduct rule to BAML primitive mapping:**
116+
117+
| Conduct rule | BAML primitive | Enforcement point |
118+
|---|---|---|
119+
| `verification.md` § Independent check | Strict return type in function signature | Parser rejects non-conforming output |
120+
| `failure-modes.md` F02 (fabrication) | `BamlValidationError` on parse fail | Calling code catches and logs |
121+
| `delegation.md` § Scope fence | Typed input parameters | Wrong input type fails at compile/call time |
122+
| `tool-use.md` § Right tool, first try | BAML function per task (one verb per function) | No ambiguity about what the function does |
123+
| `tier-sizing.md` § Prompt density | `client` declaration per tier, prompt density by tier | Top-tier client gets intent-level prompt; low-tier gets mechanical steps |
124+
125+
## Conduct propagation
126+
127+
The three patterns from [`../conduct/delegation.md`](../conduct/delegation.md) § Conduct propagation adapt to BAML's function-shaped paradigm.
128+
129+
**Pattern A — Full inherit.** Include every relevant conduct module via Jinja in the function's prompt block. Highest token cost per call; appropriate for top-tier functions or compliance-heavy domains where no module can be omitted.
130+
131+
```baml
132+
function HighStakesDecision(context: string) -> Decision {
133+
client ClaudeOpus
134+
prompt #"
135+
{% include 'conduct/discipline.md' %}
136+
{% include 'conduct/verification.md' %}
137+
{% include 'conduct/delegation.md' %}
138+
{% include 'conduct/failure-modes.md' %}
139+
140+
## Context
141+
{{ context }}
142+
143+
Decide and return a Decision with fields: verdict, rationale, risk_flags.
144+
"#
145+
}
146+
```
147+
148+
**Pattern B — Whitelist inject.** Include only the modules relevant to the specific function's task and risk profile. Recommended default; lowest per-call token cost.
149+
150+
```baml
151+
// Low-tier extraction function: only needs tool-use discipline.
152+
function ExtractEntities(text: string) -> EntityList {
153+
client ClaudeHaiku
154+
prompt #"
155+
{% include 'conduct/tool-use.md' %}
156+
157+
Extract all named entities from the text below.
158+
Text: {{ text }}
159+
"#
160+
}
161+
```
162+
163+
**Pattern C — Discovery partial.** Create a shared Jinja partial at `conduct/_index.j2` listing active modules. Functions include the partial instead of individual modules — one place to add, remove, or reorder modules.
164+
165+
```jinja2
166+
{# conduct/_index.j2 — active conduct modules for this project #}
167+
{% include 'conduct/discipline.md' %}
168+
{% include 'conduct/verification.md' %}
169+
{% include 'conduct/tool-use.md' %}
170+
```
171+
172+
```baml
173+
function ReviewPR(diff: string) -> PRReview {
174+
client ClaudeSonnet
175+
prompt #"
176+
{% include 'conduct/_index.j2' %}
177+
178+
Review the following diff.
179+
Diff: {{ diff }}
180+
"#
181+
}
182+
```
183+
184+
**Honest limit.** BAML does not document a base-function-with-overrides pattern. Propagation across functions is by shared Jinja includes and partial files, not by any class or function inheritance mechanism.
185+
186+
| Pattern | Best for | Token cost per call |
187+
|---|---|---|
188+
| Full inherit (all modules in prompt block) | Top-tier, high-stakes, compliance-heavy | High |
189+
| Whitelist inject (per-function module selection) | Mid / low-tier, bounded task scope | Medium |
190+
| Discovery partial (`conduct/_index.j2`) | Multi-function projects, consistent module set | Low–medium |
191+
192+
## Verifying the adoption
193+
194+
BAML's built-in test syntax lets you define A/B test cases inline alongside function definitions:
195+
196+
```baml
197+
test WithoutConduct {
198+
functions [ImproveCode]
199+
args {
200+
code "def f(x): return x+1"
201+
task "Review this function"
202+
}
203+
}
204+
205+
test WithConduct {
206+
functions [ImproveCode]
207+
args {
208+
code "def f(x): return x+1"
209+
task "Review this function"
210+
}
211+
// Compare output structure and quality against WithoutConduct
212+
}
213+
```
214+
215+
Run both test blocks and compare: did the conduct-loaded function produce a more structured verdict? Did it catch the missing type annotation that the bare function missed?
216+
217+
See [`../docs/self-test.md`](../docs/self-test.md) for the full fixture methodology.
218+
219+
## What this won't do
220+
221+
- Make BAML's grammar stable across versions. Function syntax, Jinja template handling, and client declaration format are evolving — verify against your current installed release before relying on specific syntax forms.
222+
- Provide multi-turn discipline. BAML is a function-call-shaped tool; conversational protections (sycophancy resistance from `doubt-engine.md`, context checkpointing from `context.md`) belong in the orchestrator that calls the BAML function, not in the BAML prompt block itself.
223+
- Cover the BAML playground, generated client code, or language-specific SDK details. This recipe operates at the `.baml` source level.
224+
- Substitute for a separate system-prompt slot. There is no equivalent in BAML; conduct text lives in the prompt template and pays token cost on every call.
225+
- Replace per-task prompt engineering. Conduct modules shift default behavior; task-specific instructions and few-shot examples still own output quality.

‎recipes/eval-harnesses.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Recipe — Eval Harnesses
2+
3+
Benchmark suite reference: nine suites mapped to agent-foundations conduct modules and failure taxonomy codes.
4+
5+
## Purpose
6+
7+
This recipe maps nine benchmark suites to the conduct modules and failure codes they exercise. Its purpose is narrow: help practitioners select the right harness for their current adoption stage without reading nine papers. It does not integrate any suite into the framework; it does not require test infrastructure beyond what each suite's own documentation specifies.
8+
9+
Use it after the enforcement layer (see [`claude-code.md`](./claude-code.md) § Enforcement wiring) gives you something deterministic to evaluate against — running these suites against a system with no enforcement hooks measures the baseline, not the improvement.
10+
11+
## Suite reference table
12+
13+
| Suite | Stars / Signal | What it tests | Framework module(s) |
14+
|-------|----------------|---------------|---------------------|
15+
| **AgentBench** | 3.4k stars (THUDM) | 8-environment LLM-as-Agent benchmark covering web, code, database, and knowledge-base tasks | Broad foundational coverage; good first baseline before targeting specific modules |
16+
| **τ-bench** | arxiv 2406.12045 | Domain-policy following under dynamic user interaction; measures whether agents respect stated rules when users push back | [`conduct/delegation.md`](../conduct/delegation.md), [`conduct/doubt-engine.md`](../conduct/doubt-engine.md) |
17+
| **τ²-bench** | ~1.1k stars (sierra-research) | Dual-control simulation: both the agent and a simulated adversarial user take turns; isolates tool-use discipline under pressure | [`conduct/tool-use.md`](../conduct/tool-use.md), [`conduct/verification.md`](../conduct/verification.md) |
18+
| **AgentDojo** | Paper | Prompt-injection robustness and untrusted-data tool execution; 97 tasks, 629 injections | [`conduct/hooks.md`](../conduct/hooks.md) enforcement; F06 (Premature action), F08 (Tool mis-invocation) |
19+
| **AgentHarm** | Paper | 110 malicious agent tasks across 11 harm categories; measures refusal, partial refusal, and recovery | F18 (Goal-conflict insider behavior), F19 (Alignment faking), F20 (Sandbagging), F21 (Weaponized tool use) |
20+
| **SYCON-Bench** | Benchmark | Multi-turn sycophancy; Turn of Flip (rate of position reversal after pushback) and Number of Flip (flip count before agent holds) | [`conduct/doubt-engine.md`](../conduct/doubt-engine.md); F01 (Sycophancy) |
21+
| **SycEval** | Paper | Progressive sycophancy ratio (agent caves over successive turns) vs. regressive (agent overcorrects when pushed); quantitative separation of the two modes | [`conduct/doubt-engine.md`](../conduct/doubt-engine.md); F01 — adds a quantitative axis the framework's qualitative doubt-engine pass lacks |
22+
| **WorkArena** | ServiceNow | Browser-based knowledge-worker workflows: form filling, record lookup, service-desk task completion | [`recipes/cursor.md`](./cursor.md); [`conduct/tool-use.md`](../conduct/tool-use.md) for browser-tool dispatch |
23+
| **Promptfoo** | Active OSS project | Custom assertion scoring: define pass/fail criteria in YAML, run against any model endpoint, compare across versions | [`conduct/verification.md`](../conduct/verification.md); lightweight alternative to full-suite benchmarks for teams in early adoption |
24+
25+
## Selection guide
26+
27+
Three practitioner profiles cover the common starting points. Pick the profile that matches your team's current priority, run the recommended suite first to establish a baseline, then expand.
28+
29+
**Enforcement-first.** Your priority is wiring the conduct rules into deterministic gates (PreToolUse / PostToolUse hooks) and confirming the gates fire correctly. Start with **Promptfoo**: define YAML assertions that mirror each conduct rule's pass condition, run against your system prompt, and iterate. Once assertions are green, run **AgentDojo** to confirm the enforcement layer holds under prompt-injection — the adversarial stress test for hooks. AgentBench as a broad baseline is the third step.
30+
31+
**Taxonomy-first.** Your priority is confirming the F-code taxonomy maps correctly to real failures your system produces. Start with **AgentHarm** (covers F18–F21) and **SYCON-Bench** (covers F01 at the turn level). These two suites together exercise the taxonomy's hardest-to-observe codes. Follow with **SycEval** to quantify the F01 surface using progressive/regressive ratios — it gives you numbers to put in `learnings.md` next to the qualitative doubt-engine log entries.
32+
33+
**Benchmark-first.** Your priority is an overall capability baseline before targeted module testing. Start with **AgentBench** (broadest environmental coverage, 3.4k stars, well-maintained). Follow with **τ-bench** to add the domain-policy and user-pushback dimension, then **τ²-bench** for adversarial dual-control. These three together cover foundational capability, policy adherence, and tool-use discipline.
34+
35+
## Open questions
36+
37+
**SYCON-Bench citation.** No arxiv ID or canonical GitHub URL was available during the research pass; the entry relies on the benchmark name alone. Confirm the canonical citation before relying on this suite in a published evaluation; add the URL to the table's Stars / Signal column when found.
38+
39+
**Datadog observability-driven harnesses.** The research pass surfaced a Datadog engineering blog post describing a DST + TLA+ + bounded model-checking harness for agent workflows. This finding is intentionally omitted from the suite table: the source is a blog post, not a specification or a public artifact, and the methodology is described at a level of abstraction that makes independent implementation impractical. If Datadog publishes a specification or open-sources the tooling, it warrants a table entry — the approach (temporal logic verification of agent traces) is directly relevant to [`conduct/verification.md`](../conduct/verification.md). Until then, treat it as an open research lead, not an adoptable harness.
40+
41+
**Suite maintenance signals.** Star counts and paper citations were current as of the research pass (2025–2026). Benchmark suites in the agent-evaluation space have a high churn rate. Before adopting any suite, confirm its repository has had activity in the last 6 months. AgentBench and τ²-bench have shown sustained maintenance; the paper-only suites (AgentDojo, AgentHarm, SycEval) should be checked more carefully.
42+
43+
**Integration with the enforcement layer.** This recipe intentionally contains no integration code. Once the framework has a canonical test-runner hookup, the suite table should be updated with framework-specific invocation examples. Until then, follow each suite's own documentation for setup.

0 commit comments

Comments
 (0)