feature: A proposal for proxy request retry.

[ggaaooppeenngg]

Feature Description

The routing logic returns a subset of potential backend vLLMs as candidates, rather than just one option, similar to a topK selection. When a proxy makes a request, it will retry among the available workers until it reaches a predefined limit (suggested limit: 3 attempts).

Why do you need this feature?

In scenarios where vLLM encounters common HTTP errors or suffers from an unrecoverable runtime error, the router can promptly redirect it to an alternative target endpoint. This ensures that the system can quickly adapt to failures and maintain service availability.

Additional context

1. Implementing an exponential backoff mechanism might violate the SLO, such as a 2s TTFT. Therefore, we should opt for a quick retry strategy with a limited number of attempts.
2. Workers can sometimes experience unrecoverable runtime errors. For example, I've encountered situations where the KV transfer buffer setting was too small, causing even a single prefill forward transfer to be blocked by the buffer availability condition. However, other endpoints may still be capable of handling the request.
3. When implementing a re-routing logic may break the boundry of the routing and proxy handling.

[YuhanLiu11]

Hey @ggaaooppeenngg thanks for bringing this up! This would be a great feature to have! Would you like to take a stab at implementing this feature?

[nejch]

For prior art, litellm does this with a num_retries option for the router: https://docs.litellm.ai/docs/proxy/config_settings#router_settings---reference

[ggaaooppeenngg]

I made a PR #421 for this issue.
I made a test server for generating errors:

from fastapi import FastAPI, HTTPException, Response, status
from fastapi.middleware.cors import CORSMiddleware
import random
import uvicorn
import asyncio

app = FastAPI(title="Abnormal Server")

# Add CORS middleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.post("/v1/chat/completions")
async def root():
    # Randomly choose between different error scenarios
    choice = random.randint(1, 2)
    print(f"Random choice: {choice}")
    if choice == 1:
        # 504 Gateway Timeout
        return Response(
            content="504 Gateway Timeout",
            status_code=status.HTTP_504_GATEWAY_TIMEOUT
        )
    elif choice == 2:
        # Raise a runtime error
        raise RuntimeError("Simulated runtime error in the server")
    elif choice == 3:
        # Simulate timeout
        await asyncio.sleep(120)
        print("mock a timeout connection")
        return {"message": "Delayed response after timeout"}

@app.get("/metrics")
async def metrics():
    # Metrics endpoint that always succeeds
    # Mock Prometheus metrics in text format
    metrics_data = """
# HELP vllm:num_requests_running Current number of running requests
# TYPE vllm:num_requests_running gauge
vllm:num_requests_running 5

# HELP vllm:num_requests_waiting Current number of waiting requests
# TYPE vllm:num_requests_waiting gauge
vllm:num_requests_waiting 12

# HELP vllm:gpu_prefix_cache_hit_rate GPU prefix cache hit rate
# TYPE vllm:gpu_prefix_cache_hit_rate gauge
vllm:gpu_prefix_cache_hit_rate 0.87

# HELP vllm:gpu_cache_usage_perc GPU cache usage percentage
# TYPE vllm:gpu_cache_usage_perc gauge
vllm:gpu_cache_usage_perc 0.62
"""
    return Response(content=metrics_data, media_type="text/plain")

if __name__ == "__main__":
    uvicorn.run("abnormal_server:app", host="0.0.0.0", port=8003)

And I setup 2 normal instances.

Based on my testing settings:

#!/bin/bash
vllm-router --routing-logic session --port 8000 --service-discovery static \
--session-key "x-user-id-no" \
--static-models Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B \
--static-backends http://localhost:8001,http://localhost:8002,http://localhost:8003

# vllm-router --routing-logic session --port 8000 --service-discovery static \
# --session-key "x-user-id" \
# --static-models Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B \
# --static-backends http://localhost:8001,http://localhost:8002,http://localhost:8003

# vllm-router --routing-logic roundrobin --port 8000 --service-discovery static \
# --static-models Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B,Qwen/Qwen3-1.7B \
# --static-backends http://localhost:8001,http://localhost:8002,http://localhost:8003

The roundrobin, least connection, and session consistent hash works fine when running the multi-qa benchmark script. The benchmark will not receive errors and keep TTFT healhy.

I found another problem that the least qps logic will constantly fall to the error endpoint (since it handles no connections), maybe leave it another issue to fix it.