mineru-router does not recover a local GPU worker after vLLM EngineDeadError because /health remains healthy

[Robin021]

Description of the bug | 错误描述

When a vLLM EngineCore used by a router-managed local mineru-api worker dies, the current parsing task fails with EngineDeadError, but the worker continues to report HTTP 200 from /health.

As a result, mineru-router continues to consider the local worker healthy and does not restart it. Every subsequent Hybrid/VLM parsing task fails immediately with the same EngineDeadError until the container or Pod is restarted manually.

This appears to be a worker lifecycle/health-reporting issue rather than a request-specific parsing issue.

Observed sequence:

1. A local GPU worker is started by mineru-router.
2. vLLM EngineCore encounters a fatal error and exits.
3. The active parsing task fails:

Hybrid processing window 1/1: pages 1-12/12 (12 pages)
Two Step Extraction:   0%|          | 0/12 [00:00<?, ?it/s]

vllm.v1.engine.exceptions.EngineDeadError:
EngineCore encountered an issue. See stack trace (above) for the root cause.

4. The HTTP process remains alive and /health still returns 200:

GET /health HTTP/1.1" 200 OK
GET /tasks/<task-id> HTTP/1.1" 200 OK

5. New parsing tasks are still routed to this worker and fail immediately.
6. Restarting the Pod restores parsing.

In AsyncTaskManager._process_task, a task exception updates only the task status:

except Exception as exc:
    task.status = TASK_FAILED
    task.error = str(exc)
    task.completed_at = utc_now_iso()
    self._signal_task_event(task_id)
    logger.exception(f"Async task failed: {task_id}")

It does not set last_worker_error, so AsyncTaskManager.is_healthy() continues to return true after EngineDeadError.

The Router already contains recovery logic that restarts a managed local worker after repeated health-check failures. That recovery path is not activated because the internal worker health endpoint remains healthy.

The same behavior is present in the mineru-3.3.1-released source:

• mineru/cli/fast_api.py: _process_task catches the exception without marking the worker unhealthy.
• mineru/cli/fast_api.py: /health relies on AsyncTaskManager.is_healthy().
• mineru/cli/router.py: the local worker is restarted after repeated non-200 health responses.

Expected behavior

After an unrecoverable inference-engine error such as vllm.v1.engine.exceptions.EngineDeadError, one of the following should happen:

1. The internal mineru-api worker marks itself unhealthy, causing /health to return 503 so mineru-router restarts it.
2. The internal worker process exits, allowing mineru-router to detect and restart it.
3. The VLM engine is recreated inside the worker.

The Router should stop routing tasks to a worker whose inference engine is permanently dead.

Suggested direction

Treat EngineDeadError as a worker-fatal error, while keeping ordinary document-specific parsing failures task-local.

For example, _process_task could record a fatal worker error when it catches EngineDeadError:

except Exception as exc:
    task.status = TASK_FAILED
    task.error = str(exc)
    task.completed_at = utc_now_iso()
    self._signal_task_event(task_id)

    if isinstance(exc, EngineDeadError):
        self.last_worker_error = str(exc)
        self._wake_waiters()

    logger.exception(f"Async task failed: {task_id}")

The exact implementation may need a backend-neutral fatal-error classification so that importing vLLM remains optional.

It would also be useful to add a regression test covering:

• a normal document parsing exception keeps the worker healthy;
• EngineDeadError makes the worker health endpoint return 503;
• mineru-router restarts a managed local worker after this failure.

How to reproduce the bug | 如何复现

Environment:

MinerU: 3.2.0
vLLM: 0.11.2
Python: 3.12
OS: Linux container on Kubernetes
Device: NVIDIA CUDA GPU
Router: mineru-router --local-gpus auto
Backend: hybrid-auto-engine

Steps:

1. Start a Router-managed local GPU worker:

mineru-router \
  --host 0.0.0.0 \
  --port 8002 \
  --local-gpus auto \
  --gpu-memory-utilization 0.7

2. Submit a PDF using the task API with backend=hybrid-auto-engine.
3. Cause or simulate a fatal vLLM EngineCore failure during inference. In production this was observed during the first Two Step Extraction request.
4. Poll the failed task and observe EngineDeadError.
5. Request /health; it still returns HTTP 200.
6. Submit another PDF. It fails immediately because the same dead engine remains attached to the worker.
7. Restart the Router container. Parsing works again.

A deterministic unit/integration reproduction can replace the predictor call with one that raises vllm.v1.engine.exceptions.EngineDeadError, then assert that the worker health endpoint changes to 503.

[dosubot]

Hi @Robin021! I'm Dosu and I'm helping the OpenDataLab team.

Your analysis is correct. The gap is confirmed in the code: _process_task catches exceptions and marks the task as failed, but does not set self.last_worker_error, so is_healthy() continues returning True [1]. The router only restarts a local worker after 5 consecutive non-200 /health responses [2] [3], which never happens in this scenario.

The is_healthy() method does already check last_worker_error — it returns False (triggering a 503) whenever that field is non-None [4]. So the infrastructure to surface fatal errors is in place; it's just that _process_task never writes to it.

One additional note: PR #4979 was opened to address a specific root cause of EngineDeadError in MinerU — a race condition where PIL images are closed before asynchronous downstream work completes under concurrent load in the hybrid-auto-engine backend. That PR copies PIL images before passing them to the predictor. If merged, it would prevent the most common trigger for this error, though it wouldn't address the general resilience gap you're describing (other fatal engine errors could still leave the worker in a zombie-healthy state).

Your suggested fix of setting last_worker_error for fatal exceptions in _process_task aligns well with how the existing health machinery works. The pattern of classifying certain exceptions as "worker-fatal" (vs. task-local) and keeping the vLLM import optional via a string check or base class would be a clean approach.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[mvanhorn]

Submitted a fix in #5188.