AutoRestTest is a complete testing software for automated API testing that combines the utility of graph theory, Large Language Models (LLMs), and multi-agent reinforcement learning (MARL) to parse the OpenAPI Specification and create enhanced comprehensive test cases. AutoRestTest specifically supports OpenAPI Specification 3.0.
Watch this demonstration video of AutoRestTest to learn how it solves complex challenges in automated REST API testing, as well as its configuration, execution steps, and output.
Note
Following the release of the demonstration video, the code base has been refactored.
Refer to this README.md for the most current setup and execution details.
The program uses LLMs for natural-language processing during the creation of the reinforcement learning tables and graph edges. AutoRestTest supports any LLM from an OpenAI-API compatible provider, including OpenAI, OpenRouter, Azure OpenAI, and local models (LocalAI, LM Studio, vLLM, Ollama, etc.).
Important
An API key from your chosen LLM provider is required. The cost per execution depends on your provider and model choice. For reference, when testing an average API with ~15 operations using GPT-4o-mini, the cost was approximately $0.1.
AutoRestTest features a modern, interactive terminal user interface built with Rich that provides:
On startup, AutoRestTest launches an interactive configuration wizard that allows you to:
- Select API specifications from discovered files or enter custom paths
- Choose LLM providers (OpenAI, OpenRouter, Local) with pre-configured model options
- Configure test duration with convenient presets (5, 10, 20, 30, 60 minutes)
- Adjust Q-learning parameters (learning rate, discount factor, exploration)
- Toggle caching options for faster repeated runs
- Override API URLs for local testing
The wizard uses sensible defaults from configurations.toml, so you can simply press Enter to accept defaults or customize any setting.
During request generation, a real-time dashboard displays:
╔══════════════════════════════════════════════════════════════════════════════╗
║ [STATUS] Request Generation ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ ┌──────────────────────────────────────────────────────────────────────┐ ║
║ │ Time Elapsed: 00:15:32 Time Remaining: 00:04:28 │ ║
║ │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.8% │ ║
║ └──────────────────────────────────────────────────────────────────────┘ ║
║ ║
║ Successfully Processed (2xx) Operations: 1,482 ║
║ Operation Coverage: 85.2% ║
║ Unique Server Errors (5xx): 7 ║
║ Total Requests Sent: 3,847 ║
║ ║
║ Status Code Distribution ║
║ ┌──────────────────────────────────────────────────────────────────────┐ ║
║ │ 200: ████████████████████████████████████████████░░░░░░░░ 1,482 │ ║
║ │ 404: ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 150 │ ║
║ │ 500: █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 7 │ ║
║ └──────────────────────────────────────────────────────────────────────┘ ║
║ ║
║ Current Operation: GET /api/users/{id} ║
║ ║
║ [COST] Total LLM usage: $0.11 USD ║
╚══════════════════════════════════════════════════════════════════════════════╝
Features include:
- Real-time progress tracking with elapsed and remaining time
- Color-coded status codes (green for 2xx, orange for 4xx, red for 5xx)
- Visual progress bars for status code distribution
- Live cost estimation based on token usage
- Current operation indicator showing what's being tested
- Mutation counter tracking the number of fuzzing mutations applied
During the Q-table initialization phase, the Value Agent and Header Agent (if enabled) perform LLM calls to generate test values for each API operation. A live progress display shows:
╭──── ⚙ Value Agent Q-Table Generation ────╮
│ │
│ ━━━━━━━━━━━━━━━━━━━━━━━─────── 67.5% │
│ │
│ Operations: 27/40 │ Elapsed: 03:45 │
│ │
│ ▶ POST /api/users │
│ │
╰───────────────────────────────────────────╯
This progress display appears for both Value Agent and Header Agent initialization, showing:
- Visual progress bar with percentage complete
- Operation count (completed/total)
- Elapsed time
- Current operation being processed
| Option | Description |
|---|---|
--skip-wizard |
Skip configuration wizard, use configurations.toml directly |
--quick |
Quick setup wizard (essential settings only) |
-s, --spec PATH |
Override specification path |
-t, --time SECONDS |
Override test duration |
--width N |
Set TUI display width (default: 100) |
Examples:
# Full interactive mode (default)
poetry run autoresttest
# Quick setup - only essential settings
poetry run autoresttest --quick
# Skip wizard entirely, use config file
poetry run autoresttest --skip-wizard
# Override spec and duration via CLI
poetry run autoresttest -s specs/original/oas/spotify.yaml -t 600We recommend using Poetry with pyproject.toml for dependency management and scripts. A poetry.lock file pins exact versions.
Steps:
- Clone the repository.
- Ensure Python 3.10.x is available (project targets
>=3.10,<3.11). - Install dependencies with Poetry (uses
poetry.lockif present):poetry install
- Create a
.envfile in the project root and add:API_KEY='<YOUR_API_KEY>'
Alternatives (provided but not recommended):
pip install -r requirements.txtconda env create -f autoresttest.yaml
Optionally, if the user wants to test specific APIs, they can install their OpenAPI Specification files within a folder
in the root directory. For convenience, we have provided a large array of OpenAPI Specifications for popular and
widely-used APIs. These Specifications can be seen in the aratrl-openapi and specs directories.
AutoRestTest includes support for running local REST API services for testing. Some services require building before use:
These services must be built before first use:
# Make the build script executable
chmod +x services/build_jdk8_1_services.sh
# Build all JDK 8_1 services
bash services/build_jdk8_1_services.shAfter building, start a service:
cd services
python3 run_service_mac.py features-service no_tokenFor detailed instructions, see FEATURES_SERVICE_SETUP.md.
Other services like genome-nexus, language-tool, youtube, etc. can be run without building:
cd services
python3 run_service_mac.py <service-name> <token>See services/README.md for the full list of supported services.
At this point, the installation step is complete and the software can be executed. However, it is important that the following configuration steps are completed for purposeful execution.
There is a wide array of configuration options available within the codebase. All configuration options are easily accessible via a single TOML file at the project root: configurations.toml.
Below are the relevant settings and where to find them in configurations.toml.
If you intend to use your own OpenAPI Specification file as described in the Installation section, set the relative path (from the project root) to that file in configurations.toml under [spec].location.
Only .yaml and .json files are supported (OpenAPI 3.0). Example: aratrl-openapi/market2.yaml.
Specification parsing is handled by Prance. If your spec has circular/self-referencing $ref chains, you can tune the resolver behavior with:
[spec].recursion_limit(default:1) — the maximum number of times a circular reference may appear in the resolution stack before a placeholder schema is substituted; a value of 1 means a self-referencing element is resolved once before being replaced.[spec].strict_validation(default:true) — whentrue, the OpenAPI spec is strictly validated and parsing stops on errors; whenfalse, invalid sections are skipped where possible so execution can continue.
Configure Q-learning parameters in configurations.toml:
[q_learning].learning_rate(default:0.1)[q_learning].discount_factor(default:0.9)[q_learning].max_exploration(epsilon; default:1, decays over time to0.1)
Instead of limiting episodes, the program limits RL iterations using a time budget. Set the duration (in seconds) under:
[request_generation].time_duration(default:1200)
Note
This time budget applies only to the MARL (Q-learning) phase. The initial value table generation phase, which runs before Q-learning begins, is not time-limited and will process all operations in the specification.
For APIs with many parameters, generating all possible combinations would cause memory exhaustion. AutoRestTest uses depth-weighted stratified sampling to bound combinations while prioritizing smaller (more diagnostic) combinations. Configure under [agent]:
| Option | Default | Description |
|---|---|---|
max_combinations |
12 |
Maximum optional parameters per combination. Required parameters are always included. |
max_total_combinations |
3000 |
Hard cap on combinations per operation. Smaller combinations are kept when truncating. |
base_samples_per_size |
200 |
Sample count at size=1; decays for larger sizes (e.g., size=5 gets ~60 samples). |
combination_seed |
42 |
Seed for reproducible random sampling. Change for different samples. |
Example distribution for an operation with 30 parameters:
- Size 1-2: All combinations enumerated (30 + 435 = 465)
- Size 3-12: Random samples (~700 total)
- Total: ~1,200 unique combinations (well under 3K cap)
Tip
The above steps alter the variables across all agents used within the software. If the user desires to change
the individual agent parameters, they can navigate to the src/autoresttest/agents files and change the parameters.
The Value Agent's Q-table initialization can be parallelized using a thread pool, which significantly speeds up startup for APIs with many operations. Configure this under [agent.value]:
[agent.value].parallelize(default:true) — whentrue, operations are processed concurrently using a thread pool; whenfalse, the original sequential depth-first traversal is used.[agent.value].max_workers(default:4) — number of worker threads for parallel generation (ignored ifparallelizeisfalse). Set to match your CPU core count for optimal performance.
Note
Rate-limited responses (HTTP 429) are automatically retried with exponential backoff.
AutoRestTest supports any LLM from an OpenAI-API compatible provider. Configure the model and provider in configurations.toml:
| Option | Default | Description |
|---|---|---|
[llm].engine |
gpt-4o-mini |
Model identifier. Format depends on provider (e.g., gpt-4o-mini, google/gemini-2.0-flash-001, anthropic/claude-3-haiku). |
[llm].api_base |
https://api.openai.com/v1 |
API endpoint URL. Change for alternative providers. |
[llm].creative_temperature |
1 |
Temperature for creative parameter generation. |
[llm].strict_temperature |
1 |
Temperature for repair or deterministic flows. |
[llm].max_tokens |
30000 |
Maximum tokens for LLM response. Set to -1 to omit and use provider default. |
Example configurations:
# OpenAI (default)
[llm]
engine = "gpt-4o-mini"
api_base = "https://api.openai.com/v1"
# OpenRouter (access to many models)
[llm]
engine = "google/gemini-2.0-flash-001"
api_base = "https://openrouter.ai/api/v1"
# Local model (LM Studio, Ollama, etc.)
[llm]
engine = "local-model"
api_base = "http://localhost:1234/v1"Warning
The model must support JSON mode. Most recent models from major providers support this feature. The console output will list token usage for analyzing costs, if supplied by the provider.
The software can cache the graph and Q-tables to reduce cost and speed up repeated runs. Configure this behavior under:
[cache].use_cached_graph(default:true)[cache].use_cached_table(default:true)
Important
The cache parameter structure changed on Dec. 10, 2025. Recreate caches (clear cache/ contents) so they remain compatible with newer runs.
Note
When enabled, these options store and read cached data under the cache/ directory at the project root (for example, cache/graphs/ and cache/q_tables/).
If disk usage becomes a concern, you can delete the cached project information in cache/ after use; the data will be regenerated on the next run when needed.
AutoRestTest contains an optional Header agent responsible for testing the API with different authentication headers. Due to difficulties of different authentication flows, the Header agent is only able to use Basic Authentication. By default, the agent is disabled.
Toggle via configurations.toml:
[agents.header].enabled(default:false)
Caution
The Header agent Q-table should be rerun when executing services with local databases that refresh, as the user credentials may become invalid.
For APIs that require custom headers (such as API keys or Bearer tokens), you can configure static headers in the [custom_headers] section. These headers are included with every request.
[custom_headers]
X-API-Key = "your-static-key"
Authorization = "Bearer ${ACCESS_TOKEN}"Environment variable interpolation is supported using the ${VAR_NAME} syntax. Store sensitive values in your .env file:
ACCESS_TOKEN=your_bearer_token
Caution
If you enable the Header Agent ([agents.header].enabled = true) alongside custom headers, be aware that the Header Agent may override your Authorization header during testing. The Header Agent uses Basic Authentication tokens and has higher priority than custom static headers. For Bearer token or API key authentication, keep the Header Agent disabled.
By default, AutoRestTest extracts the API base URL from the servers field in your OpenAPI specification. If you need to target a different host (e.g., a local development server or staging environment), you can override this behavior in the [api] section:
[api]
override_url = true
host = "localhost"
port = 8080When override_url is set to true, the tool constructs the API URL as http://{host}:{port}/ instead of using the specification's server URL.
| Option | Default | Description |
|---|---|---|
override_url |
false |
When false, uses the URL from the OpenAPI spec. When true, uses the custom host and port. |
host |
localhost |
The hostname for the custom API URL. |
port |
8080 |
The port number for the custom API URL. |
Tip
This is useful when testing local services that run on a different port than specified in the OpenAPI spec, or when the spec contains a production URL but you want to test against a local or staging environment.
Run the script using Poetry, after following the installation instructions:
poetry run autoresttest
To indicate the specification for execution, set [spec].location in configurations.toml. This path must be relative to the project root.
For ease of use, the software can be executed using Docker. The user can apply the following commands from the root directory sequentially to execute AutoRestTest using the pre-built Docker image:
docker build -t autoresttest .
docker run -it autoresttest
Ensure that configurations.toml is configured correctly before executing the software.
Throughout AutoRestTest's execution, the TUI provides real-time visual feedback including:
- Phase indicators showing current execution stage (Graph Construction, Q-Table Initialization, Request Generation)
- Live dashboard during request generation with time tracking, status code distribution, and cost estimation
- Final report with summary statistics, success rates, and token usage
- Smart status code grouping - automatically groups by category (2xx, 4xx, 5xx) when there are many different codes
- Color-coded time indicators - remaining time changes color based on percentage remaining (red < 5%, yellow < 20%)
AutoRestTest will generate a series of files in the data directory within the root folder for easy access to execution results.
The following are the files generated:
report.json: Contains the compiled report of the API testing process, including key summary statistics and information.operation_status_codes.json: Contains the distribution of status codes for each operation.q_tables.json: Contains the completed Q-tables for each agent.server_errors.json: Contains the unique server errors encountered during the testing process. Only JSON-seriable errors are stored.successful_parameters.json: Contains the successful parameters for each operation.successful_responses.json: Contains the successful responses for each operation.successful_bodies.json: Contains the successful object request bodies for each operation.successful_primitives.json: Contains the successful primitive request bodies for each operation.
These files contain the necessary information for analysis into the success of AutoRestTest.
Note
The output files can grow in size according to the number of operations and the duration of execution.
For example, the file containing successful responses can grow to be several gigabytes when executing for a long duration.
It is recommended to clear the data directory when the files are no longer needed.