You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #605 (fix/embedding-space-pin-602), which pinned the query path and made a dim mismatch fail loud in lib/memory retrieval. That stops the silent degrade-to-lexical, but the root asymmetry upstream remains and is out of scope for the surgical fix:
app/api/embed/route.ts produces vectors at different dims depending on backend: OpenAI text-embedding-3-small with dimensions: 512 vs local nomic-embed-text at 768. There is no single declared CORPUS_EMBED_DIM the route pins to.
Beyond-the-fix work to make the space a real contract on ALL paths:
Declare one authoritative CORPUS_EMBED_DIM (and model id) as estate config; the embed route stamps the produced space (model + dim) into the response / stored entry metadata.
Persist the embedding space alongside each stored vector (MemoryEntry) so a mismatch is detectable without inference from the first entry.
Re-embed migration: when the active backend's space differs from the stored corpus space, re-embed (or refuse writes into a mixed store) instead of accumulating mixed-dim entries.
Mirror the same PinnedSpace check in any OTHER consumer of /api/embed (graph/sae/tune paths) — the estate rule is pin on ALL paths incl. callbacks.
Refs #605, #602, prophet-workspace#82, prophet-workspace#76. cc @mdheller
Follow-up to #605 (fix/embedding-space-pin-602), which pinned the query path and made a dim mismatch fail loud in
lib/memoryretrieval. That stops the silent degrade-to-lexical, but the root asymmetry upstream remains and is out of scope for the surgical fix:app/api/embed/route.tsproduces vectors at different dims depending on backend: OpenAItext-embedding-3-smallwithdimensions: 512vs localnomic-embed-textat 768. There is no single declaredCORPUS_EMBED_DIMthe route pins to.NOETICA_EMBED_MODEL) silently produces a corpus embedded in a different space than later queries. fix(memory): pin corpus embedding dim on the query path — fail loud, never degrade to lexical (#602/#82) #605 now catches this at query time by raising — good (fail-closed), but the user experience is a hard error rather than a repaired corpus.Beyond-the-fix work to make the space a real contract on ALL paths:
CORPUS_EMBED_DIM(and model id) as estate config; the embed route stamps the produced space (model + dim) into the response / stored entry metadata.MemoryEntry) so a mismatch is detectable without inference from the first entry./api/embed(graph/sae/tune paths) — the estate rule is pin on ALL paths incl. callbacks.Refs #605, #602, prophet-workspace#82, prophet-workspace#76. cc @mdheller