Skip to content

fix(wiki): prevent concurrent duplicate page identities - #2850

Merged
lyingbug merged 3 commits into
Tencent:mainfrom
JYao-Chen:feat/wiki-page-canonicalization
Aug 28, 2026
Merged

fix(wiki): prevent concurrent duplicate page identities#2850
lyingbug merged 3 commits into
Tencent:mainfrom
JYao-Chen:feat/wiki-page-canonicalization

Conversation

@JYao-Chen

Copy link
Copy Markdown
Contributor

Description

Wiki ingest allows concurrent batches for the same knowledge base and protects Reduce with per-slug locks. Two map workers can still extract the same same-type display title with different romanized slugs before either page exists. Because the reducers lock different slug keys, both pages can be created.

This change closes that window by:

  • resolving an exact same-type, normalized-title candidate deterministically before the semantic dedup LLM call;
  • reserving the chosen slug in Redis by (knowledge base, page type, normalized title) so concurrent batches converge before summaries and Reduce are materialized;
  • using a batch-local claim in Lite mode and when Redis is unavailable;
  • coalescing candidates that converge to one slug while preserving aliases, cited chunks, and the richer fallback text;
  • keeping punctuation significant, so 寓言 and 《寓言》 remain distinct identities, and keeping entity/concept namespaces separate.

The claim is temporary and does not add a schema migration or change existing page data.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

No linked issue.

Testing

  • go test ./internal/application/service -count=1
  • go test -race ./internal/application/service -run TestWikiIdentityClaimConvergesDifferentSlugs -count=1
  • go vet ./internal/application/service
  • git diff --check origin/main...HEAD

The added regression tests cover concurrent workers proposing different slugs for the same title, authoritative existing-page resolution, Lite-mode batch convergence, entity/concept separation, punctuation-sensitive work titles, and evidence preservation during coalescing.

Checklist

  • git diff --check origin/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped golangci-lint (not installed in the development environment)
  • Full changed-package tests were run
  • Self-reviewed the code
  • Added/updated tests covering the change
  • No documentation update required
  • No breaking changes

Screenshots / Recordings

Not applicable; backend-only change.

JYao-Chen and others added 3 commits August 28, 2026 13:18
Only exact same-title pages may overwrite a Redis identity claim. Successful
Redis reads now refresh the batch-local map so concurrent batches stay
aligned, citation slugs are reclaimed, and Reduce remaps onto the claimed
slug before locking.
Exact title resolution was one seq-scan per extracted item, and a dirty Redis claim could split concurrent workers. Batch the lookup, memoize it per batch, overwrite invalid claims, and keep remapped Item.Slug in sync.
@lyingbug
lyingbug merged commit 2523dc7 into Tencent:main Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants