ArkSim: Simulating Multi-Turn Users to Evaluate LlamaIndex Agents

[Yaoa4]

Hi all — wanted to share something we’ve been working on that might be useful for folks building agents with LlamaIndex.

We recently open-sourced ArkSim, a synthetic user simulator designed to evaluate LLM agents through multi-turn interactions.

One gap we kept running into: most evaluation setups focus on single-turn outputs or static datasets, but many real issues only show up over longer interactions.

What ArkSim does

ArkSim simulates diverse user behaviors and runs multi-turn conversations against your agent, helping surface:

• failures across full workflows (not just individual steps)
• edge cases from more realistic user behavior
• regressions when changing prompts, tools, or models

LlamaIndex integration

We put together a simple example using LlamaIndex agents here:
https://github.com/arklexai/arksim/tree/main/examples/integrations/llamaindex

The goal is to make it easy to plug into existing agent setups and start stress-testing interactions.

We’re still exploring a few things and would love input from the community. Appreciate any thoughts or feedback — especially from folks working on agents reliability and evaluation.