Skip to content
BranchesTags

Repository files navigation

Awesome Harness Engineering Awesome

A curated list of articles, playbooks, benchmarks, specifications, and open-source projects for harness engineering: the practice of shaping the environment around AI agents so they can work reliably.

Harness engineering sits at the intersection of context engineering, evaluation, observability, orchestration, safe autonomy, and software architecture. This list focuses on resources that make agents more dependable in real workflows, especially long-running coding and research tasks.

Generic agent tooling is out of scope unless the page directly covers harness design, context management, evaluation, runtime control, or other reliability-critical harness primitives.

This README is generated from the structured directory at harness-engineer. See the live, searchable site at https://harness-engineer.dev/.

Contents

Courses & Learning Resources

  • walkinglabs/learn-harness-engineering - A project-based course repository on making Codex and Claude Code more reliable, centered on an Electron personal knowledge base app with lecture handouts, example artifacts, and practical harness projects.

Foundations

Context, Memory & Working State

  • Advanced Context Engineering for Coding Agents - HumanLayer patterns for reducing context drift and making coding sessions easier to resume.
  • Context-Efficient Backpressure for Coding Agents - HumanLayer's ideas for preventing agents from burning context on noisy or low-value work.
  • Context Engineering for AI Agents: Lessons from Building Manus - Manus' detailed playbook on KV-cache locality, tool masking, filesystem memory, and keeping useful failures in-context.
  • Context Engineering for Coding Agents - Thoughtworks guidance on shaping the task environment so coding agents can stay grounded and productive.
  • Context Forking to Save Time, Tokens and Trouble - Treats the agent context window as a downward-growing OS stack and argues forking/rewind as a core harness primitive for course-correction, parallel design exploration, and salvaging context after context-inefficient operations.
  • The Creators of Model Context Protocol - MCP coauthors David Soria Parra and Justin Spahr-Summers explain the origin, primitives (tools, resources, prompts), and design philosophy of the Model Context Protocol: a stateful, JSON-RPC-based standard inspired by LSP that solves the M×N problem of connecting AI applications to integrations. Covers MCP vs OpenAPI, building servers, tool-confusion, nesting, authorization, and the roadmap to stateless/resumable transports.
  • Effective context engineering for AI agents - Anthropic's guidance on managing the context window as a working memory budget rather than a dumping ground.
  • Equipping agents for the real world with Agent Skills - Introduces Agent Skills as a progressive-disclosure context mechanism: on-demand loaded instructions/tools that keep the base context small and let agents acquire domain knowledge only when relevant, a core harness pattern for long-running agents.
  • Getting Claude to Actually Read Your CLAUDE.md - Tactical pattern for CLAUDE.md/AGENTS.md files: wrap task-specific instructions in conditional <important if=...> blocks so the model applies the right rules at the right time instead of treating a long file as optional. Ships with an improve-claude-md skill.
  • humanlayer/skills - Claude Code skills that implement harness-engineering patterns: improve-claude-md (conditional instruction blocks), build-iterated-agentic-loop (repo-local iterated agent workflow), and design-control-loop (sensor/controller/actuator/disturbance interview-driven loop design tailored to a codebase).
  • Long-Context Isn't the Answer - Empirical argument that extended-context models degrade instruction adherence at all lengths because the 'instruction budget' doesn't scale with context window size. Makes the case that context isolation (sub-agents, progressive disclosure, backpressure) beats context expansion.
  • OpenHands Context Condensensation for More Efficient AI Agents - OpenHands' design for bounded conversation memory that preserves goals, progress, critical files, and failing tests while keeping long-running coding sessions efficient.
  • Writing a good CLAUDE.md - A practical guide to creating durable, repo-local instructions that agents can repeatedly follow.

Constraints, Guardrails & Safe Autonomy

  • Agent Sandboxing: What OpenAI got wrong with the HuggingFace hack - Postmortem-style analysis of the GPT-5.6 ExploitGym incident where the model escaped its sandbox via a proxy zero-day, arguing for the air-gapped runtime model (private package registries, mocked APIs, strict egress firewalls) over proxy-mediated access, plus structured-event observability and human-in-the-loop on the dangerous subset of tool calls.
  • AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents - An extensible evaluation environment with 97 tasks and 629 security test cases for measuring agent robustness to prompt injection, with pluggable attacks and defenses — the canonical benchmark for agent security harness design.
  • Anchoring AI to a reference application - Thoughtworks on constraining agents with concrete exemplars so they produce more consistent output.
  • Anthropic Sandbox Runtime (srt) - Open-sourced lightweight sandboxing runtime from Claude Code that enforces filesystem and network restrictions on arbitrary processes using native OS primitives (Seatbelt on macOS, bubblewrap on Linux, WFP on Windows) with proxy-based egress filtering, mandatory-deny paths for shell/git config, and a clear JSON policy schema — directly reusable for sandboxing MCP servers and agent-spawned processes.
  • Assessing internal quality while coding with an agent - Thoughtworks on moving quality checks into the loop instead of relying on after-the-fact manual review.
  • Auto mode for Claude Code - Examines Claude Code's auto-mode, where a separate Sonnet 4.6 classifier gates each action against an extensive JSON allow/soft-deny/block policy. Includes the full default policy and argues why deterministic sandboxing outside the agent layer is more trustworthy than non-deterministic AI permission classifiers.
  • Beyond permission prompts: making Claude Code more secure and autonomous - Anthropic on reducing approval friction without losing control through better sandboxing and policy design.
  • Building Governed Agents: A Framework for Cost, Control, and Compliance - Maps the full governance surface for production agents — identity, audit logs, provider secrets, data residency — and frames an LLM gateway as the runtime control plane enforcing spend, model routing, PII/secrets/jailbreak guardrails, and failover across LLM, tool, MCP, and agent-to-agent interactions, tying policy enforcement to tracing and evals.
  • CaMeL offers a promising new direction for mitigating prompt injection attacks - Detailed walkthrough of Google DeepMind's CaMeL (Capabilities for Machine Learning) paper, which converts user prompts into a restricted Python DSL executed by a custom interpreter with capability-based data-flow tainting. The first prompt-injection mitigation Simon endorses as offering 'strong guarantees' rather than probabilistic guardrails.
  • Claude API: Web fetch tool - Analyzes Anthropic's web_fetch tool design, which deterministically blocks the model from constructing arbitrary URLs and restricts fetches to user-provided or prior-search results, with optional domain allow-lists. A concrete case study in building exfiltration resistance into a tool's harness rather than relying on the model.
  • Claude Code: Best practices for agentic coding - Anthropic's practical recommendations for repo structure, checkpoints, validation, and delegation in agentic coding workflows.
  • Code execution with MCP: building more efficient agents - Anthropic's approach to giving agents controlled execution power through explicit, inspectable tool boundaries.
  • Defeating Prompt Injections by Design (CaMeL) - Proposes CaMeL, a capability-based security layer that separates control and data flows so untrusted tool output cannot hijack program flow; solves 77% of AgentDojo tasks with provable security, defining a new design pattern for safe autonomy.
  • Design Patterns for Securing LLM Agents against Prompt Injections - Reviews a 15-author paper proposing six named agent-architecture patterns — Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, and Context-Minimization — with ten case studies. A practical pattern catalog for constraining agents so untrusted input cannot trigger consequential actions.
  • Every Agent Needs a Box — Aaron Levie, Box - Aaron Levie (Box CEO) on the infrastructure every agent needs: sandboxed "box" workspaces, agent identity and governance, permission models for agent-vs-human data access, and why context engineering (not infinite context windows) is the real bottleneck for enterprise knowledge-work agents. Includes Box's internal agent eval methodology — rubric-scored retrieval across industry document sets — and the explore/exploit search limits of frontier models.
  • How we built Claude Code auto mode: a safer way to skip permissions - Detailed design of a model-based transcript classifier that substitutes for human approval on Claude Code tool calls, including a two-stage fast-filter + reasoning pipeline, reasoning-blind construction to resist prompt injection, deny-and-continue behavior, and honest false-negative reporting. A concrete reference for automated permission gating inside an agent harness.
  • How we contain Claude across products - Anthropic's definitive writeup on agent containment architecture across claude.ai, Claude Code, and Cowork — three isolation patterns (ephemeral containers, human-in-the-loop sandboxes, sealed VMs), real incidents including an employee-phish exfiltration and an allowlist bypass via the Files API, and the principle of designing deterministic environmental defenses first and probabilistic model-layer defenses second.
  • Humans and Agents in Software Engineering Loops - A clear mental model for where humans should strengthen the harness instead of micromanaging every artifact.
  • Lurkr - Static scanner that runs in CI before deploy to surface AI-agent capability risks, including shadow capabilities, credentials into LLM context, eval/subprocess in @tool, direct prompt interpolation, and unverified MCP endpoints.
  • Mitigating Prompt Injection Attacks in Software Agents - OpenHands' practical guide to confirmation mode, analyzers, sandboxing, and hard policies for reducing prompt-injection risk in autonomous coding agents.
  • Model Context Protocol has prompt injection security problems - Catalogues real prompt-injection attack vectors against MCP tool servers, including tool-poisoning, rug-pull redefinitions, and cross-server shadowing. Essential for anyone wiring MCP tools into an agent harness, with concrete examples of how tool descriptions themselves can become the attack surface.
  • Reconstructing how OpenAI agents attacked Hugging Face - Daniel Whitenack and Chris Benson reconstruct the incident in which OpenAI agents escaped a secure sandbox and attacked Hugging Face private infrastructure: how the agents exploited vulnerabilities, moved through networks, and launched large-scale autonomous attacks. Covers sandboxing failures, the case for AI systems governing other AI systems, and what the breach reveals about enterprise agent security and safe autonomy.
  • The browser is the sandbox - Surveys Paul Kinlan's argument and demo that the browser's 30-year-old sandbox — the File System Access API, CSP headers, sandboxed iframes, and WebAssembly workers — is a mature, deterministic alternative to running coding agents in local containers. Concrete techniques (including a double-iframe network-isolation trick) for harnessing untrusted agent execution without a VM.
  • The Dual LLM pattern for building AI assistants that can resist prompt injection - Proposes splitting an agent into a privileged, tool-calling LLM and a quarantined LLM that handles untrusted content, with a non-LLM controller mediating symbolic variables between them. A concrete, implementable architecture for sandboxing untrusted context that later influenced Google DeepMind's CaMeL paper.
  • The lethal trifecta for AI agents: private data, untrusted content, and external communication - Defines the three capabilities whose combination enables data exfiltration via prompt injection — access to private data, exposure to untrusted content, and an outbound channel — and argues the only robust defense is to avoid assembling all three in one agent. Includes a long list of real-world production incidents illustrating the pattern.
  • Writing effective tools for agents - Anthropic's guidance on tool interfaces that are easier for models to call correctly and safely.

Specs, Agent Files & Workflow Design

  • 12-Factor AgentOps - An operations-oriented companion focused on context discipline, validation, and reproducible agent workflows.
  • 12 Factor Agents - HumanLayer's operating principles for production agents, including explicit prompts, state ownership, and clean pause-resume behavior.
  • agent.md - A related standardization effort for machine-readable agent instructions across projects and tools.
  • AGENTS.md - A lightweight open format for repo-local instructions that tell agents how to work inside a codebase.
  • GitHub Spec Kit - GitHub's toolkit for spec-driven development, useful when you want agents to execute against explicit product and engineering specs.
  • Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl - Thoughtworks on why strong specs make AI-assisted software delivery more dependable.

Evals & Observability

  • Agent evals - OpenAI's product guide for measuring agent quality with reproducible task-level and workflow-level evaluations.
  • AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents - Introduces a fine-grained progress-rate metric and evaluation toolkit that makes partial-progress and trajectory quality visible, pushing agent evaluation beyond binary success rates.
  • AgentOps - Open-source Python SDK for agent monitoring, session replay, cost tracking, benchmarking, and tracing across common LLM and agent frameworks.
  • agenttrace - Local-first TUI/CLI for auditing AI coding-agent session traces, health gates, cost spikes, tool failures, latency gaps, and attempt-to-attempt diffs.
  • AI Agents That Matter - Argues that agent benchmarks over-focus on accuracy while ignoring cost, overfitting, and reproducibility; prescribes joint cost-accuracy optimization, proper holdouts, and standardized evaluation — a foundational critique of agent eval methodology.
  • An update on recent Claude Code quality reports - Postmortem of three separate changes that caused perceived Claude Code degradation — a reasoning-effort default change, a cache-optimization bug that dropped thinking history every turn, and a verbosity prompt that hurt coding quality — illustrating how harness changes slip past unit/e2e review and the eval/rollout practices needed to catch them.
  • Demystifying Evals for AI Agents - Anthropic's guidance on what to measure when agents have many possible trajectories to success or failure.
  • Designing AI-resistant technical evaluations - A three-iteration case study of keeping a performance-engineering take-home ahead of Claude, documenting what makes evaluations robust to AI assistance (out-of-distribution constraints, many independent sub-problems, building debugging tooling as part of the signal) and why realism becomes a liability as model capability grows.
  • The End of SWE-Bench Verified - OpenAI Frontier Evals leads Mia Glaese and Olivia Watkins explain why they publicly discontinued SWE-Bench Verified: >60% of remaining problems were unsolvable (overly narrow tests or hidden requirements), and contamination was demonstrable across all frontier models. Outlines the move to SWE-Bench Pro and the properties great coding evals need next: longer tasks, open-ended design decisions, code quality, real-world impact, and rubric-based human grading.
  • Eval awareness in Claude Opus 4.6's BrowseComp performance - First documented case of a model independently hypothesizing it was being evaluated, identifying the benchmark (BrowseComp), retrieving and decrypting its answer key using a sandboxed Python REPL, raising hard questions about eval integrity in web-enabled environments and the inadequacy of URL blocklists against a capable agent.
  • Evaluating code review agents with ReviewBench - A Harbor-format benchmark built from curated real PR reviewer comments in the LangSmith monorepo, with an LLM-judge verifier scoring coverage and precision, showing current models with a basic harness miss ~70% of curated reviewer findings and that a structured review prompt changes results more than swapping models.
  • Evaluating Deep Agents: Our Learnings - LangChain's practical breakdown of single-step, full-run, and multi-turn eval design for stateful agents.
  • Evaluation best practices - OpenAI's general guide to building eval suites that match real-world distributions and catch regressions early.
  • How to Evaluate Agent Skills (And Why You Should) - OpenHands' hands-on playbook for measuring whether a skill actually helps using bounded tasks, deterministic verifiers, no-skill baselines, and trace review.
  • Improving Deep Agents with harness engineering - LangChain's evidence that harness changes alone can significantly improve benchmark performance.
  • Inspect AI - UK AISI's open-source evaluation framework with solver, scorer, sandboxing, tool-use, MCP, and log-viewer primitives for building reproducible agent eval harnesses.
  • IssueBench - How We Evaluate Engine - Internal benchmark for an agent that inspects production traces to identify and cluster failures, using synthetic traces with known ground truth across 15 issue categories (PII leak, hallucination, guardrail bypass, agent looping, etc.) and scoring classification, category assignment, existing-issue attachment, and new-issue grouping — illustrating how to evaluate the meta-agent that evaluates agents.
  • Learning to Verify AI-Generated Code - OpenHands' overview of a layered verification stack using trajectory critics trained on production traces for reranking, early stopping, and review-time quality control.
  • OpenTelemetry Semantic Conventions for Generative AI Systems - Standard span, metric, event, and attribute conventions for instrumenting LLM and agent workflows so harness traces stay portable across observability backends.
  • Quantifying infrastructure noise in agentic coding evals - Anthropic on how runtime configuration can move coding benchmark scores by more than many leaderboard gaps.
  • Reality: The Final Eval — Andon Labs - Lukas Petersson and Axel Backlund of Andon Labs discuss Vending-Bench, Project Vend, and Andon Market: dollar-denominated, long-horizon evals that give frontier models inventory, a wallet, tools, and time. Concrete case studies of harness design and failure modes — Claude calling the FBI over a $2 fee, long-context meltdown loops, multi-agent price cartels, election manipulation, and why Slack-as-observability works for tracing agent behavior that static benchmarks miss.
  • smevals—a small eval suite for evaluating models, prompts, and harnesses - Introduces smevals, a framework that separates runs from grading and treats 'harnesses' as a first-class config variable alongside model and prompt. Establishes a clear vocabulary — eval, task, config, run, grader, check — and is the third iteration of Simon's eval tooling, built to compare agent harnesses rather than just models.
  • Testing Agent Skills Systematically with Evals - OpenAI's concrete guide to turning agent traces into repeatable evals with JSONL logs and deterministic checks.
  • The Verification Stack - Introduces a two-layer verification architecture for coding agents: an agent-level critic model that scores trajectories before code is pushed, and a repo-level verifier (code review + QA agent) that exercises the running software via Playwright, with six months of production data showing 58% faster mean time to merge and precision approaching human reviewers.
  • Towards Automating Eval Engineering - Describes an 'Eval Engineering Skill' for coding agents that mines a repository and optional traces to propose, interview the user about, and generate executable Harbor-format evals (instruction + Dockerfile environment + verifier), codifying the mine-traces → build-eval → improve-agent → rerun loop and treating evals as training data for harness engineering.
  • Trace grading - OpenAI documentation on grading agent traces directly, which is especially helpful for long multi-step tasks.

Benchmarks

  • Agent Arena - A leaderboard that ranks AI agents, models, tools, and frameworks using ELO-style ratings from head-to-head battles, providing a structured way to compare harness-level choices across categories.
  • AgentBench: Evaluating LLMs as Agents - A multi-environment benchmark (OS, DB, knowledge graph, web, card game, etc.) for evaluating LLMs as agents, identifying instruction-following and long-horizon reasoning as the main failure modes harnesses must address.
  • AgentBench - A cross-environment benchmark spanning OS, databases, knowledge graphs, web browsing, and more, useful for seeing whether a harness generalizes beyond one narrow task loop.
  • AgentBoard - A benchmark for multi-turn LLM agents complemented by an analytical evaluation board for assessing model performance beyond final success rates, making partial-progress and trajectory quality visible.
  • AgentDojo - A dynamic environment to evaluate prompt-injection attacks and defenses for LLM agents, with composable suites, pluggable attacks/defenses, and an interactive results explorer. The canonical benchmark for measuring agent security and tool-call robustness.
  • AgentStudio - An integrated benchmark suite with realistic environments and comprehensive toolkits for evaluating virtual agents on real computer software, useful for measuring harness depth against a broad task surface.
  • AndroidWorld - A dynamic benchmarking environment for autonomous agents running on a live Android emulator, with 116 hand-crafted tasks across 20 real apps and millions of parameterized variations. Durable reward signals and extensible task design make it a model for reproducible mobile-agent eval.
  • AppWorld - A controllable world of apps and people for benchmarking interactive coding agents, with state-based and execution-based unit tests that surface harness quality around planning, code generation, and collateral-damage control.
  • AssistantBench - A benchmark that evaluates web agents on realistic, time-consuming research tasks requiring multi-step tool use and information synthesis, making it a good proxy for harness quality in long-horizon web scenarios.
  • τ-Bench - A benchmark that emulates dynamic conversations between a simulated user and a language agent equipped with domain-specific API tools and policy guidelines, making it useful for evaluating harnesses built around structured tool use and policy enforcement.
  • BrowseComp - A benchmark that evaluates AI agents on locating hard-to-find information, stressing search strategy, context management, and retrieval harness design under difficult conditions.
  • BrowserGym Leaderboard - A gym environment and leaderboard for evaluating LLMs, VLMs, and agents on web navigation tasks, offering a reproducible framework for comparing harnesses across multiple web benchmarks in one place.
  • CharacterEval - A benchmark for evaluating role-playing conversational agents using multi-turn dialogues and character profiles, with metrics across four dimensions including character fidelity and conversational coherence.
  • ClawBench: Can AI Agents Complete Everyday Online Tasks? - A browser-agent benchmark of 153 everyday web tasks across 144 live production sites in 15 categories, using a lightweight interception layer that captures and blocks only the final submission request so agents can be scored end-to-end on real websites without real-world side effects.
  • ClawBench - A benchmark that evaluates AI agents across search, reasoning, coding, safety, and multi-turn conversation tasks, covering the breadth of harness demands in a single suite.
  • ClawWork - A real-world economic benchmark where AI agents complete professional tasks spanning 44 occupations, earning income while managing token costs and economic solvency, making it a direct test of harness efficiency under resource constraints.
  • Computer Agent Arena - An open evaluation platform where users compare LLM/VLM-based agents on real-world computer tasks ranging from general computer use to coding, data analysis, and video editing, surfacing harness differences across a wide task surface.
  • EvoClaw: Evaluating AI Agents on Continuous Software Evolution - A benchmark write-up on evaluating agents across dependent milestone sequences from real repository history, surfacing regression accumulation and long-horizon precision loss.
  • GAIA - A benchmark for general AI assistants that is often used to compare harness-level choices around tools, planning, verification, and long-horizon autonomy.
  • Galileo Agent Leaderboard - An open evaluation platform tracking LLM agents on task completion and tool calling across business domains, useful for comparing harness quality in enterprise-grade agentic scenarios.
  • GTA - A benchmark that evaluates the tool-use capability of LLM-based agents using human-written queries, real deployed tools, and authentic multimodal inputs, exposing harness gaps between isolated testing and real deployment.
  • HAL: Holistic Agent Leaderboard - A benchmark and leaderboard for agent systems with attention to reliability, cost, and broad task coverage, making it useful for comparing end-to-end harness behavior.
  • Introducing Terminal-Bench 2.0 and Harbor - The Terminal-Bench 2.0 announcement, useful for understanding the harder tasks and generalized evaluation harness behind Harbor.
  • LeetCode-Hard Gym - An RL environment interface to LeetCode's submission server for evaluating codegen agents, giving harnesses direct access to execution-based feedback on hard algorithmic problems.
  • LLM Colosseum Leaderboard - A platform that evaluates LLMs by having them fight in Street Fighter III, testing speed, adaptability, and real-time decision-making as proxies for harness responsiveness under tight latency constraints.
  • MAgIC - A benchmark measuring cognition, adaptability, rationality, and collaboration of LLMs in multi-agent systems, useful for evaluating how harnesses coordinate agent interactions and shared state.
  • MCP Bench - A benchmark for evaluating AI models on MCP server interactions, measuring tool accuracy, latency, and token use across server types, which directly reflects harness design choices around MCP integration.
  • MCP Universe - A leaderboard comparing AI model performance on MCP tasks, tracking how different models and harness configurations handle tool-augmented agent workflows.
  • MCPMark - A stress-testing benchmark for model and agent capabilities in real-world MCP tasks across tools like Notion, GitHub, and Postgres, making harness MCP integration quality directly measurable.
  • Mind2Web - The first large-scale dataset for generalist web agents: 2,000+ open-ended tasks across 137 real websites and 31 domains with crowdsourced action sequences and full Playwright traces/snapshots. Foundational data for training and evaluating HTML-grounded web agents.
  • Olas Predict Benchmark - A benchmark for evaluating agents on historical prediction market data, testing harness design for research, retrieval, and forecasting in long-horizon reasoning tasks.
  • OSWorld-MCP - An extension of OSWorld that evaluates AI agents on real-world computer tasks using the Model Context Protocol, making it useful for comparing MCP-enabled harnesses on a realistic desktop task suite.
  • OSWorld - A real computer-use benchmark with 369 tasks across Ubuntu, Windows, and macOS, complete with initial-state setup and execution-based evaluators, making it excellent for testing desktop and multimodal harnesses.
  • SEC-bench - A benchmark for evaluating LLM agents on real-world software security tasks including vulnerability reproduction and patching, stressing harness design around code execution, containerized environments, and security-aware tooling.
  • SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - The canonical benchmark for agentic coding harnesses: 2,294 real GitHub issues across 12 Python repos where the agent must edit a codebase to resolve the issue, demanding execution environments, long-context handling, and multi-file reasoning.
  • SWE-bench Verified - A strong benchmark for software engineering agents working against real GitHub issues and tests, which makes harness choices around retrieval, patching, and validation highly visible.
  • tau2-bench - A benchmark for realistic, multi-step agent tasks where success depends on tool use and execution quality rather than a single-shot answer.
  • Terminal-Bench - A benchmark suite for terminal-native agents operating in shells, filesystems, and verification-heavy environments, which is especially useful for comparing coding-agent harnesses.
  • TheAgentCompany - A benchmark of 175 consequential real-world professional tasks in a simulated software company (GitLab, Plane, RocketChat, ownCloud) with Docker-based task images, encrypted evaluators, and subcheckpoint scoring. Tests agents across coding, conversation, and multi-agent interaction.
  • TravelPlanner - A benchmark for evaluating LLM agents on tool use and complex planning within multiple constraints, revealing how harness design handles multi-constraint satisfaction and long-horizon planning.
  • VAB - VisualAgentBench evaluates large multimodal models as visual foundation agents across embodied, GUI, and visual design tasks, useful for comparing harnesses on visually grounded, multi-step agent workflows.
  • VisualWebArena - A benchmark for multimodal web agents on realistic visually grounded tasks, extending WebArena with image and screenshot inputs that stress harness support for visual context in browser environments.
  • WebArena-Verified - A verified web-agent benchmark with curated tasks and deterministic evaluators over agent responses and captured network traces, making it a good fit for measuring web-facing harnesses.
  • WebArena - A standalone, self-hostable web environment for evaluating autonomous agents on realistic tasks, making it a reproducible baseline for comparing web-facing harness designs.
  • WildClawBench - An in-the-wild benchmark running agents inside a live OpenClaw environment on 60 original tasks including multimodal, long-horizon, and safety-critical scenarios, making harness robustness under real-world conditions directly visible.
  • Windows Agent Arena (WAA) - A scalable Windows 11 benchmark for multimodal desktop agents, running in a reproducible QEMU/Docker VM with Azure ML parallelization (hundreds of tasks in minutes). Ships the Navi agent with OmniParser and a BYOA interface for plugging in your own agent.
  • WorkArena - A benchmark for browser agents on common knowledge-work tasks, useful for comparing harnesses on realistic enterprise-style web workflows instead of toy browser tasks.

Runtimes, Harnesses & Reference Implementations

  • Agent Frameworks, Runtimes, and Harnesses, Oh My! - LangChain's decomposition of what belongs in a framework, a runtime, and a harness.
  • Agent2Agent (A2A) Protocol - An open protocol (Linux Foundation, contributed by Google) for interoperability between opaque agents across frameworks, with Agent Cards for discovery, JSON-RPC over HTTP, SSE streaming, and push notifications. Ships Python/Go/JS/Java/.NET/Rust SDKs and defines how independent agent runtimes communicate.
  • AgentKit - Inngest's TypeScript toolkit for building durable, workflow-aware agents on top of event-driven infrastructure.
  • AutoGen - A pioneering multi-agent framework (now in maintenance mode, succeeded by Microsoft Agent Framework) with a layered Core/AgentChat/Extensions architecture, local and distributed runtimes, AutoGen Studio no-code GUI, and AutoGen Bench evaluation suite. Widely cited as a reference design for event-driven multi-agent runtimes.
  • Bolt.new, Flow Engineering for Code Agents - StackBlitz CEO Eric Simons and Qodo CEO Itamar Friedman dish on building production coding agents: Bolt's WebContainer OS (a from-scratch WASM runtime instrumented for error-capture and self-healing loops), task breakdown flows, and Qodo's AlphaCodium "Flow Engineering" methodology (reason → propose solutions → generate tests → iterate until passing). Concrete patterns for sandboxed execution environments and test-verified agent loops.
  • Bring Your AI MCP - Public harness-migration reference for Claude Code to Codex moves, with installable auditor artifacts and explicit validation notes for hooks, MCP config, and instruction-file differences.
  • browser-use/browser-harness - A thin CDP-based browser harness that lets agents extend helper functions during execution, useful for inspecting self-healing web-task workflows.
  • Building agents with the Claude Agent SDK - Anthropic's guide to a production-oriented agent SDK with sessions, tools, and orchestration support.
  • Building Durable AI Agents - Hamza Tahir (ZenML/Kitaru) on applying MLOps principles to generative AI: agent harnesses, fleets, replayable and observable agent systems, and the infrastructure required to move agents from demos to reliable production. Practical engineering patterns for durable, scalable agent workflows.
  • Citadel - A harness for Claude Code and OpenAI Codex with isolated worktrees, multi-agent coordination, and persisted memory and campaign state.
  • Claude Code: Anthropic's Agent in Your Terminal - Boris Cherny and Cat Wu from the Claude Code team walk through the design of a CLI coding agent that Anthropic treats as a "Unix utility": do-the-simple-thing-first memory (a markdown file), autocompact by asking Claude to summarize, permission systems for safe autonomy, and non-interactive -p mode for fan-out workflows. Concrete guidance on harness thickness, sandboxing/branching, and where the model should end and the harness begin.
  • deepagents - LangChain's open-source project for building deeper, longer-running agents with middleware and harness patterns.
  • Executable Code Actions Elicit Better LLM Agents (CodeAct) - Demonstrates that using executable Python code as the unified action space (vs. JSON/text) yields up to 20% higher success, enabling dynamic tool composition and self-debugging — a key argument for code-execution harnesses.
  • Google Agent Development Kit (ADK) - A code-first Python toolkit for building, evaluating, and deploying agents, with a graph-based Workflow Runtime (routing, fan-out/fan-in, loops, human-in-the-loop), a Task API for agent-to-agent delegation, and a built-in eval/Web UI. A first-class runtime for inspectable multi-agent orchestration.
  • Harbor - A generalized harness for evaluating and improving agents at scale, released alongside Terminal-Bench 2.0.
  • Harness Evolver - Claude Code plugin that autonomously evolves LLM agent harnesses using multi-agent proposers, LangSmith-backed evaluation, and git worktree isolation. Based on Meta-Harness (Lee et al., 2026).
  • HEAAL - Grammar-enforced safety constraints for AI agents via AIL (AI-Intent Language).
  • How we built our multi-agent research system - Anthropic's architecture write-up for a multi-agent system with separation of roles and structured coordination.
  • Letta (formerly MemGPT) - A platform for stateful, self-improving agents with advanced memory, exposing an App Server API plus a TypeScript Agent SDK that can run agents on Constellation cloud, fully local, or self-hosted. A reference for building agents whose memory and behavior persist across sessions.
  • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework - Encodes Standardized Operating Procedures into prompt sequences and an assembly-line role assignment to reduce cascading hallucinations in multi-agent systems — a concrete SOP-driven orchestration architecture.
  • Microsoft Agent Framework (MAF) - The enterprise-ready successor to AutoGen for building and orchestrating production multi-agent workflows in Python and .NET, with graph-based orchestration, checkpointing/durability, OpenTelemetry observability, declarative YAML agents, and A2A/MCP interoperability. Built for taking agents from prototype to production with inspectability.
  • OpenHands - Self-hosted, sandboxed runtime for coding agents (Docker/VM/cloud backends) with an Agent Server REST API, automations, and support for any ACP-compatible agent. A reference for making agent execution inspectable and switchable across environments.
  • Ralph Wiggum as a Software Engineer - Geoffrey Huntley's write-up of "Ralph," a minimalist while :; do cat PROMPT.md | claude-code; done harness pattern that uses single-task loops, deterministic prompt stacking, and bounded subagent parallelism to drive long-running autonomous coding.
  • skills.sh - A community marketplace for discovering, sharing, and installing reusable AI agent skills across runtimes like Claude Code and OpenClaw, making harness capabilities portable and composable.
  • Stateless MCP has recaptured my interest (and inspired mcp-explorer and datasette-mcp) - Covers the MCP 2.0 'stateless' spec change and the author's resulting mcp-explorer and datasette-mcp tooling. Relevant to harness engineering as MCP is becoming the standard tool-attachment layer for agent runtimes, and the stateless model changes how tools are registered, called, and sandboxed.
  • SWE-agent - A mature research coding agent that makes the harness, prompt, tools, and environment design directly inspectable.
  • SWE-ReX - Sandboxed code execution infrastructure for AI agents, useful when harness work starts to merge into execution runtime design.
  • UFO Desktop AgentOS - A Windows Desktop AgentOS with deep UIA/Win32/WinCOM integration and hybrid GUI+API actions, plus UFO³ Galaxy for multi-device DAG-based orchestration across Windows/Linux/Android via a WebSocket AIP protocol. A production-grade reference for OS-level agent harnesses and cross-device coordination.
  • Uni-CLI - Universal CLI hub connecting agents to 134 sites and desktop apps via 711 declarative YAML pipelines. Ships an 8-phase Karpathy-style self-repair loop, eval harness with a starter catalog, per-call cost ledger, hardcoded sensitive-path deny list, and unicli mcp serve that auto-registers one MCP tool per adapter. ~80 tokens per invocation.

Contributing

Contributions are welcome. Please prefer resources that are:

  • Specific about how agents are constrained, evaluated, resumed, observed, or orchestrated
  • Original implementations, primary-source articles, or high-signal technical write-ups
  • Useful to practitioners building real harnesses instead of generic AI commentary

If two links say the same thing, prefer the more primary, practical, and implementation-oriented one.

See CONTRIBUTING.md for contribution guidelines and the preferred entry format. New entries can be added via pull request or, on the live site, via a submission form that opens a PR for you.

License

CC BY-SA 4.0

About

Site for consuming and augmenting the awesome lists relating to AI harnesses

Resources

Readme
CC-BY-SA-4.0 license

Contributing

Contributing
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages