Fix floret vectors StringStore sharing after Vocab unpickling (E018 with nlp.pipe n_process>1)#13994
Open
joakimwar wants to merge 1 commit into
Open
Conversation
In unpickle_vocab, vocab.vectors was assigned while vocab.strings was still the fresh Vocab's empty StringStore; assigning vocab.strings afterwards left the vectors table holding a reference to the discarded store. Strings interned after unpickling (e.g. by the tokenizer in an nlp.pipe(n_process>1) spawn worker) were then missing from vectors.strings, so floret-mode vectors raised E018 on string lookup in Vectors.get_batch. Set vocab.strings before vocab.vectors so the vectors setter shares the vocab's StringStore with the vectors table, and extend test_pickle_vocab to assert the sharing survives a pickle round-trip. Fixes explosion#13993. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #13993:
nlp.pipe(texts, n_process>1)immediately and deterministically raises[E018](wrapped in[E871]) for any pipeline whosevocab.vectors.mode == "floret"(e.g. the officialsv_core_news_lg), under thespawnmultiprocessing start method (the default on macOS and Windows).Cause: floret-mode vectors resolve each token hash back to its literal text via
self.stringsinVectors.get_batch(). Normallyvectors.stringsis the sameStringStoreobject asvocab.strings— theVocab.vectorssetter explicitly shares it. Butunpickle_vocabassignedvocab.vectors = vectorsbeforevocab.strings = sstore, so the setter shared the freshVocab()'s empty StringStore, which the very next line replaced. After unpickling — which is exactly how a spawnednlp.pipeworker receives the pipeline — the vectors table holds a reference to a discarded, empty store. Every string the tokenizer interns in the worker goes intovocab.strings, the floret lookup consults the orphaned store, and the first lookup raises E018.Fix: assign
vocab.stringsbeforevocab.vectorsinunpickle_vocab, so the existing setter logic shares the correct store. No new mechanism, just an ordering fix.Testing (macOS arm64, Python 3.10, source build of
master):pickle.loads(pickle.dumps(nlp.vocab)),vocab.strings is vocab.vectors.stringsisFalsebefore this fix andTrueafter.sv_core_news_lg,batch_size=40,n_process=2): unpatched build fails with the exact E018/hash from the issue; patched build processes all 1000 docs.spacy/tests/vocab_vectorsandspacy/tests/serializepass (222 passed).test_pickle_vocab(which already uses floret-mode vectors) to assert that the unpickled vectors share the vocab's StringStore and that a string interned after unpickling resolves throughvectors.strings. The new assertions fail on unpatched spaCy 3.8.7 and pass with this fix.(🤖 Claude Code was used to investigate the bug and prepare this fix.)
Types of change
Bug fix.
Checklist