Skip to content

Commit 54d13bb

Browse files
kdeleekdelee
committed
docs: Add and expand guides for CLI, config, and data formats
Created new guides for configuration options (environment variables, .env files) and supported data formats. It also updates the main CLI reference and links to the new guides.
1 parent 79d91d3 commit 54d13bb

File tree

3 files changed

+106
-8
lines changed

3 files changed

+106
-8
lines changed

‎docs/guides/cli.md

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,34 @@ This page provides a reference for the `guidellm` command-line interface. For mo
44

55
## `guidellm benchmark run`
66

7-
This command is the primary entrypoint for running benchmarks.
7+
This command is the primary entrypoint for running benchmarks. It has many options that can be specified on the command line or in a scenario file.
88

9-
### Target Configuration
9+
### Scenario Configuration
10+
11+
| Option | Description |
12+
| --- | --- |
13+
| `--scenario <PATH or NAME>` | The name of a builtin scenario or path to a scenario configuration file. Options specified on the command line will override the scenario file. |
14+
15+
### Target and Backend Configuration
1016

1117
These options configure how `guidellm` connects to the system under test.
1218

1319
| Option | Description |
1420
| --- | --- |
15-
| `--target <URL>` | **Required.** The endpoint of the target system, e.g., `http://localhost:8080`. |
16-
| `--target-header <HEADER>` | A header to send with requests to the target. This option can be specified multiple times to send multiple headers. The header should be in the format `"Header-Name: Header-Value"`. For example: `--target-header "Authorization: Bearer my-secret-token"` |
17-
| `--target-skip-ssl-verify` | A flag to disable SSL certificate verification when connecting to the target. This is useful for development environments with self-signed certificates, but should be used with caution in production. |
21+
| `--target <URL>` | **Required.** The endpoint of the target system, e.g., `http://localhost:8080`. Can also be set with the `GUIDELLM__OPENAI__BASE_URL` environment variable. |
22+
| `--target-header <HEADER>` | A header to send with requests to the target. Can be specified multiple times. Example: `--target-header "Authorization: Bearer my-secret-token"`. |
23+
| `--target-skip-ssl-verify` | A flag to disable SSL certificate verification when connecting to the target. |
24+
| `--backend-type <TYPE>` | The type of backend to use. Defaults to `openai_http`. |
25+
| `--model <NAME>` | The ID of the model to benchmark within the backend. |
26+
27+
### Data and Request Configuration
28+
29+
These options define the data to be used for benchmarking and how requests will be generated.
30+
31+
| Option | Description |
32+
| --- | --- |
33+
| `--data <SOURCE>` | The data source. This can be a HuggingFace dataset ID, a path to a local data file, or a synthetic data configuration. See the [Data Formats Guide](./data_formats.md) for more details. |
34+
| `--rate-type <TYPE>` | The type of request generation strategy to use (e.g., `constant`, `poisson`, `sweep`). |
35+
| `--rate <NUMBER>` | The rate of requests per second for `constant` or `poisson` strategies, or the number of steps for a `sweep`. |
36+
| `--max-requests <NUMBER>` | The maximum number of requests to run for each benchmark. |
37+
| `--max-seconds <NUMBER>` | The maximum number of seconds to run each benchmark for. |

‎docs/guides/configuration.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,24 +19,40 @@ For example, to set the `api_key` for the `openai` backend, you would use the fo
1919
export GUIDELLM__OPENAI__API_KEY="your-api-key"
2020
```
2121

22-
### Target Configuration
22+
### Target and Backend Configuration
2323

2424
You can configure the connection to the target system using environment variables. This is an alternative to using the `--target-*` command-line flags.
2525

2626
| Environment Variable | Description | Example |
2727
| --- | --- | --- |
28-
| `GUIDELLM__OPENAI__HEADERS` | A JSON string representing a dictionary of headers to send to the target. These headers will override any default headers (like `Authorization` from `api_key`). | `export GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token", "X-Custom-Header": "value"}'` |
28+
| `GUIDELLM__OPENAI__BASE_URL` | The endpoint of the target system. Equivalent to the `--target` CLI option. | `export GUIDELLM__OPENAI__BASE_URL="http://localhost:8080"` |
29+
| `GUIDELLM__OPENAI__API_KEY` | The API key to use for bearer token authentication. | `export GUIDELLM__OPENAI__API_KEY="your-secret-api-key"` |
30+
| `GUIDELLM__OPENAI__BEARER_TOKEN` | The full bearer token to use for authentication. | `export GUIDELLM__OPENAI__BEARER_TOKEN="Bearer your-secret-token"` |
31+
| `GUIDELLM__OPENAI__HEADERS` | A JSON string representing a dictionary of headers to send to the target. These headers will override any default headers. | `export GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token"}'` |
2932
| `GUIDELLM__OPENAI__ORGANIZATION` | The OpenAI organization to use for requests. | `export GUIDELLM__OPENAI__ORGANIZATION="org-12345"` |
3033
| `GUIDELLM__OPENAI__PROJECT` | The OpenAI project to use for requests. | `export GUIDELLM__OPENAI__PROJECT="proj-67890"` |
3134
| `GUIDELLM__OPENAI__VERIFY_SSL` | Set to `false` or `0` to disable SSL certificate verification. | `export GUIDELLM__OPENAI__VERIFY_SSL=false` |
35+
| `GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS` | The default maximum number of tokens to request for completions. | `export GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=2048` |
36+
37+
### General HTTP Settings
38+
39+
These settings control the behavior of the underlying HTTP client.
40+
41+
| Environment Variable | Description |
42+
| --- | --- |
43+
| `GUIDELLM__REQUEST_TIMEOUT` | The timeout in seconds for HTTP requests. Defaults to 300. |
44+
| `GUIDELLM__REQUEST_HTTP2` | Set to `true` or `1` to enable HTTP/2 support. Defaults to true. |
45+
| `GUIDELLM__REQUEST_FOLLOW_REDIRECTS` | Set to `true` or `1` to allow the client to follow redirects. Defaults to true. |
46+
3247

3348
### Using a `.env` file
3449

3550
You can also place these variables in a `.env` file in your project's root directory:
3651

3752
```dotenv
3853
# .env file
54+
GUIDELLM__OPENAI__BASE_URL="http://localhost:8080"
3955
GUIDELLM__OPENAI__API_KEY="your-api-key"
4056
GUIDELLM__OPENAI__HEADERS='{"Authorization": "Bearer my-token"}'
4157
GUIDELLM__OPENAI__VERIFY_SSL=false
42-
```
58+
```

‎docs/guides/data_formats.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Data Formats
2+
3+
The `--data` argument for the `guidellm benchmark run` command accepts several different formats for specifying the data to be used for benchmarking.
4+
5+
## Local Data Files
6+
7+
You can provide a path to a local data file in one of the following formats:
8+
9+
- **CSV (.csv)**: A comma-separated values file. The loader will attempt to find a column with a common name for the prompt (e.g., `prompt`, `text`, `instruction`).
10+
- **JSON (.json)**: A JSON file. The structure should be a list of objects, where each object represents a row of data.
11+
- **JSON Lines (.jsonl)**: A file where each line is a valid JSON object.
12+
- **Text (.txt)**: A plain text file, where each line is treated as a separate prompt.
13+
14+
If the prompt column cannot be automatically determined, you can specify it using the `--data-args` option:
15+
```bash
16+
--data-args '{"text_column": "my_custom_prompt_column"}'
17+
```
18+
19+
## Synthetic Data
20+
21+
You can generate synthetic data on the fly by providing a configuration string or file.
22+
23+
### Configuration Options
24+
25+
| Parameter | Description |
26+
| --- | --- |
27+
| `prompt_tokens` | **Required.** The average number of tokens for the generated prompts. |
28+
| `output_tokens` | **Required.** The average number of tokens for the generated outputs. |
29+
| `samples` | The total number of samples to generate. Defaults to 1000. |
30+
| `source` | The source text to use for generating the synthetic data. Defaults to a built-in copy of "Pride and Prejudice". |
31+
| `prompt_tokens_stdev` | The standard deviation of the tokens generated for prompts. |
32+
| `prompt_tokens_min` | The minimum number of text tokens generated for prompts. |
33+
| `prompt_tokens_max` | The maximum number of text tokens generated for prompts. |
34+
| `output_tokens_stdev` | The standard deviation of the tokens generated for outputs. |
35+
| `output_tokens_min` | The minimum number of text tokens generated for outputs. |
36+
| `output_tokens_max` | The maximum number of text tokens generated for outputs. |
37+
38+
### Configuration Formats
39+
40+
You can provide the synthetic data configuration in one of three ways:
41+
42+
1. **Key-Value String:**
43+
```bash
44+
--data "prompt_tokens=256,output_tokens=128,samples=500"
45+
```
46+
47+
2. **JSON String:**
48+
```bash
49+
--data '{"prompt_tokens": 256, "output_tokens": 128, "samples": 500}'
50+
```
51+
52+
3. **YAML or Config File:**
53+
Create a file (e.g., `my_config.yaml`):
54+
```yaml
55+
prompt_tokens: 256
56+
output_tokens: 128
57+
samples: 500
58+
```
59+
And use it with the `--data` argument:
60+
```bash
61+
--data my_config.yaml
62+
```

0 commit comments

Comments
 (0)