File tree Expand file tree Collapse file tree 1 file changed +5
-5
lines changed Expand file tree Collapse file tree 1 file changed +5
-5
lines changed Original file line number Diff line number Diff line change @@ -10,23 +10,23 @@ The image can be used to run OpenAI compatible server and is available on Docker
10
10
``` bash
11
11
docker run --runtime nvidia --gpus all \
12
12
-v ~ /.cache/huggingface:/root/.cache/huggingface \
13
- --env " HUGGING_FACE_HUB_TOKEN=<secret> " \
13
+ --env " HUGGING_FACE_HUB_TOKEN=$HF_TOKEN " \
14
14
-p 8000:8000 \
15
15
--ipc=host \
16
16
vllm/vllm-openai:latest \
17
- --model mistralai/Mistral-7B-v0.1
17
+ --model Qwen/Qwen3-0.6B
18
18
```
19
19
20
20
This image can also be used with other container engines such as [ Podman] ( https://podman.io/ ) .
21
21
22
22
``` bash
23
- podman run --gpus all \
23
+ podman run --device nvidia.com/gpu= all \
24
24
-v ~ /.cache/huggingface:/root/.cache/huggingface \
25
25
--env " HUGGING_FACE_HUB_TOKEN=$HF_TOKEN " \
26
26
-p 8000:8000 \
27
27
--ipc=host \
28
- vllm/vllm-openai:latest \
29
- --model mistralai/Mistral-7B-v0.1
28
+ docker.io/ vllm/vllm-openai:latest \
29
+ --model Qwen/Qwen3-0.6B
30
30
```
31
31
32
32
You can add any other [ engine-args] ( ../configuration/engine_args.md ) you need after the image tag (` vllm/vllm-openai:latest ` ).
You can’t perform that action at this time.
0 commit comments