Skip to content

fix(milvus): handle concurrent database creation race on first boot - #3876

Open
AdalbertoDam wants to merge 2 commits into
HKUDS:mainfrom
AdalbertoDam:fix/milvus-first-boot-database-race
Open

fix(milvus): handle concurrent database creation race on first boot#3876
AdalbertoDam wants to merge 2 commits into
HKUDS:mainfrom
AdalbertoDam:fix/milvus-first-boot-database-race

Conversation

@AdalbertoDam

Copy link
Copy Markdown
Contributor

Description

Fixes a crash-on-first-boot (MilvusException: database already exist,
Milvus server error code 65535) that any multi-process LightRAG deployment
(Kubernetes replicas > 1, or any second instance started concurrently
against the same fresh Milvus) can hit when
MilvusVectorDBStorage._create_milvus_client
races on Milvus's non-atomic list_databases()-then-create_database()
check.

Related Issues

Fixes #3875

Changes Made

  • MilvusVectorDBStorage._create_milvus_client now wraps
    client.create_database(db_name) in a try/except MilvusException,
    treating a case-insensitive "already exist" substring match against the
    exception's message as benign, and re-raising any other MilvusException
    unchanged. The match is on the message rather than MilvusException.code
    because the numeric code for this condition isn't consistently assigned
    across Milvus server versions, and re-verifying via a second
    list_databases() call risks observing stale cluster metadata and
    re-raising even though the create actually succeeded.
  • Added four regression tests to tests/kg/milvus_impl/test_milvus_index_creation.py:
    • test_create_database_race_already_exists_is_swallowed — a
      create_database call raising an "already exist" MilvusException
      must not fail _create_milvus_client().
    • test_create_database_race_message_match_is_case_insensitive — the
      match tolerates the server's varying capitalization of the message.
    • test_create_database_unrelated_milvus_exception_is_reraised — any
      other MilvusException (e.g. a permission error) from
      create_database must still propagate, so the fix narrowly targets the
      known race and doesn't swallow genuine errors.
    • test_create_database_non_milvus_exception_is_not_swallowed — a
      non-MilvusException error (e.g. a connection reset) is never
      inspected for the race message and always propagates unmodified.

Checklist

  • Changes tested locally
  • Code reviewed
  • Documentation updated (if necessary) — N/A, no behavior/API change on
    the non-race path
  • Unit tests added (if applicable)

Additional Notes

Root cause: list_databases() then create_database() is not atomic on
the Milvus server. Any deployment starting more than one LightRAG process
against the same fresh Milvus database name can have two processes both
observe the database as absent and both call create_database(); the loser
receives error code 65535 ("database already exist"), which currently
propagates uncaught through initialize_storages() and kills that process.
On any later restart the database already exists, so the race window only
appears on a genuinely fresh volume.

Verification: reproduced independently using the official
ghcr.io/hkuds/lightrag:latest image against a fresh, official Milvus standalone
stack (two replicas started at once) — one replica crashed with the exact traceback
this PR fixes, while the surviving replica (identical image/config) booted cleanly,
ruling out a configuration issue. Re-verified against the patched code (uv run lightrag-server, two processes, same fresh Milvus): one process hit the
identical MilvusException the "before" run crashed on, and both processes
now complete Application startup complete. and return 200 from
/health. ruff check / ruff format --check pass on both changed files,
and tests/kg/milvus_impl/test_milvus_index_creation.py passes in full (46
tests, including the 4 new ones).

Fixes a crash-on-first-boot (MilvusException: database already exist)
that any multi-process LightRAG deployment can hit when
MilvusVectorDBStorage._create_milvus_client races on Milvus's
non-atomic list_databases()-then-create_database() check.
@danielaskdd

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T10:12:34.752201Z 523c466 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 523c4666cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/kg/milvus_impl/test_milvus_index_creation.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Milvus first-boot database-creation race

2 participants