Skip to content

Fix floret vectors StringStore sharing after Vocab unpickling (E018 with nlp.pipe n_process>1)#13994

Open
joakimwar wants to merge 1 commit into
explosion:masterfrom
joakimwar:fix/floret-vectors-multiprocessing-unpickle
Open

Fix floret vectors StringStore sharing after Vocab unpickling (E018 with nlp.pipe n_process>1)#13994
joakimwar wants to merge 1 commit into
explosion:masterfrom
joakimwar:fix/floret-vectors-multiprocessing-unpickle

Conversation

@joakimwar

Copy link
Copy Markdown

Description

Fixes #13993: nlp.pipe(texts, n_process>1) immediately and deterministically raises [E018] (wrapped in [E871]) for any pipeline whose vocab.vectors.mode == "floret" (e.g. the official sv_core_news_lg), under the spawn multiprocessing start method (the default on macOS and Windows).

Cause: floret-mode vectors resolve each token hash back to its literal text via self.strings in Vectors.get_batch(). Normally vectors.strings is the same StringStore object as vocab.strings — the Vocab.vectors setter explicitly shares it. But unpickle_vocab assigned vocab.vectors = vectors before vocab.strings = sstore, so the setter shared the fresh Vocab()'s empty StringStore, which the very next line replaced. After unpickling — which is exactly how a spawned nlp.pipe worker receives the pipeline — the vectors table holds a reference to a discarded, empty store. Every string the tokenizer interns in the worker goes into vocab.strings, the floret lookup consults the orphaned store, and the first lookup raises E018.

Fix: assign vocab.strings before vocab.vectors in unpickle_vocab, so the existing setter logic shares the correct store. No new mechanism, just an ordering fix.

Testing (macOS arm64, Python 3.10, source build of master):

  • Verified the severed reference directly: after pickle.loads(pickle.dumps(nlp.vocab)), vocab.strings is vocab.vectors.strings is False before this fix and True after.
  • A/B against the issue's repro (1000 Swedish docs, sv_core_news_lg, batch_size=40, n_process=2): unpatched build fails with the exact E018/hash from the issue; patched build processes all 1000 docs.
  • spacy/tests/vocab_vectors and spacy/tests/serialize pass (222 passed).
  • Extended the existing test_pickle_vocab (which already uses floret-mode vectors) to assert that the unpickled vectors share the vocab's StringStore and that a string interned after unpickling resolves through vectors.strings. The new assertions fail on unpatched spaCy 3.8.7 and pass with this fix.

(🤖 Claude Code was used to investigate the bug and prepare this fix.)

Types of change

Bug fix.

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

In unpickle_vocab, vocab.vectors was assigned while vocab.strings was
still the fresh Vocab's empty StringStore; assigning vocab.strings
afterwards left the vectors table holding a reference to the discarded
store. Strings interned after unpickling (e.g. by the tokenizer in an
nlp.pipe(n_process>1) spawn worker) were then missing from
vectors.strings, so floret-mode vectors raised E018 on string lookup in
Vectors.get_batch.

Set vocab.strings before vocab.vectors so the vectors setter shares the
vocab's StringStore with the vectors table, and extend test_pickle_vocab
to assert the sharing survives a pickle round-trip.

Fixes explosion#13993.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nlp.pipe(n_process>1) crashes with [E018]/[E871] for any pipeline using floret-mode vectors (e.g. sv_core_news_lg)

1 participant