Skip to content

Declare a single corpus embedding-space contract in the embed route + re-embed migration on backend switch (#602/#82 follow-up) #606

Description

@mdheller

Follow-up to #605 (fix/embedding-space-pin-602), which pinned the query path and made a dim mismatch fail loud in lib/memory retrieval. That stops the silent degrade-to-lexical, but the root asymmetry upstream remains and is out of scope for the surgical fix:

  • app/api/embed/route.ts produces vectors at different dims depending on backend: OpenAI text-embedding-3-small with dimensions: 512 vs local nomic-embed-text at 768. There is no single declared CORPUS_EMBED_DIM the route pins to.
  • Consequence: switching backends (add/remove an OpenAI key, change NOETICA_EMBED_MODEL) silently produces a corpus embedded in a different space than later queries. fix(memory): pin corpus embedding dim on the query path — fail loud, never degrade to lexical (#602/#82) #605 now catches this at query time by raising — good (fail-closed), but the user experience is a hard error rather than a repaired corpus.

Beyond-the-fix work to make the space a real contract on ALL paths:

  1. Declare one authoritative CORPUS_EMBED_DIM (and model id) as estate config; the embed route stamps the produced space (model + dim) into the response / stored entry metadata.
  2. Persist the embedding space alongside each stored vector (MemoryEntry) so a mismatch is detectable without inference from the first entry.
  3. Re-embed migration: when the active backend's space differs from the stored corpus space, re-embed (or refuse writes into a mixed store) instead of accumulating mixed-dim entries.
  4. Mirror the same PinnedSpace check in any OTHER consumer of /api/embed (graph/sae/tune paths) — the estate rule is pin on ALL paths incl. callbacks.

Refs #605, #602, prophet-workspace#82, prophet-workspace#76. cc @mdheller

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions