fix(milvus): handle concurrent database creation race on first boot - #3876
Open
AdalbertoDam wants to merge 2 commits into
Open
fix(milvus): handle concurrent database creation race on first boot#3876AdalbertoDam wants to merge 2 commits into
AdalbertoDam wants to merge 2 commits into
Conversation
Fixes a crash-on-first-boot (MilvusException: database already exist) that any multi-process LightRAG deployment can hit when MilvusVectorDBStorage._create_milvus_client races on Milvus's non-atomic list_databases()-then-create_database() check.
Collaborator
|
@codex review |
Contributor
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 523c4666cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes a crash-on-first-boot (
MilvusException: database already exist,Milvus server error code
65535) that any multi-process LightRAG deployment(Kubernetes
replicas > 1, or any second instance started concurrentlyagainst the same fresh Milvus) can hit when
MilvusVectorDBStorage._create_milvus_clientraces on Milvus's non-atomic
list_databases()-then-create_database()check.
Related Issues
Fixes #3875
Changes Made
MilvusVectorDBStorage._create_milvus_clientnow wrapsclient.create_database(db_name)in atry/except MilvusException,treating a case-insensitive
"already exist"substring match against theexception's message as benign, and re-raising any other
MilvusExceptionunchanged. The match is on the message rather than
MilvusException.codebecause the numeric code for this condition isn't consistently assigned
across Milvus server versions, and re-verifying via a second
list_databases()call risks observing stale cluster metadata andre-raising even though the create actually succeeded.
tests/kg/milvus_impl/test_milvus_index_creation.py:test_create_database_race_already_exists_is_swallowed— acreate_databasecall raising an "already exist"MilvusExceptionmust not fail
_create_milvus_client().test_create_database_race_message_match_is_case_insensitive— thematch tolerates the server's varying capitalization of the message.
test_create_database_unrelated_milvus_exception_is_reraised— anyother
MilvusException(e.g. a permission error) fromcreate_databasemust still propagate, so the fix narrowly targets theknown race and doesn't swallow genuine errors.
test_create_database_non_milvus_exception_is_not_swallowed— anon-
MilvusExceptionerror (e.g. a connection reset) is neverinspected for the race message and always propagates unmodified.
Checklist
the non-race path
Additional Notes
Root cause:
list_databases()thencreate_database()is not atomic onthe Milvus server. Any deployment starting more than one LightRAG process
against the same fresh Milvus database name can have two processes both
observe the database as absent and both call
create_database(); the loserreceives error code
65535("database already exist"), which currentlypropagates uncaught through
initialize_storages()and kills that process.On any later restart the database already exists, so the race window only
appears on a genuinely fresh volume.
Verification: reproduced independently using the official
ghcr.io/hkuds/lightrag:latestimage against a fresh, official Milvus standalonestack (two replicas started at once) — one replica crashed with the exact traceback
this PR fixes, while the surviving replica (identical image/config) booted cleanly,
ruling out a configuration issue. Re-verified against the patched code (
uv run lightrag-server, two processes, same fresh Milvus): one process hit theidentical
MilvusExceptionthe "before" run crashed on, and both processesnow complete
Application startup complete.and return200from/health.ruff check/ruff format --checkpass on both changed files,and
tests/kg/milvus_impl/test_milvus_index_creation.pypasses in full (46tests, including the 4 new ones).