Allow updating existing documents by removing duplicates before upload

[Copilot]

Currently, uploading the same document twice creates duplicated chunks in the vector database. This PR implements deduplication functionality that removes all existing chunks with the same filename before adding new ones, allowing for document updates without duplication.

Changes Made

Azure CosmosDB NoSQL Vector Store Path

• Added deletion by SQL filter before adding new documents
• Uses SQL query SELECT * FROM c WHERE c.metadata.source = "{filename}" to find existing documents
• Includes error handling for cases where the container doesn't exist yet (first upload)
• Properly escapes filenames containing quotes to prevent SQL injection

FAISS Vector Store Path (Local Development)

• Added removeDuplicateDocuments() helper function to find and delete existing documents
• Iterates through the document mapping to find documents with matching metadata.source
• Deletes documents by their IDs using the FAISS delete() method
• Only runs when an existing FAISS store is loaded

Implementation Details

Before this change:

// Azure CosmosDB path
await AzureCosmosDBNoSQLVectorStore.fromDocuments(documents, embeddings, { credentials });

// FAISS path  
const store = await FaissStore.load(faissStoreFolder, embeddings);
await store.addDocuments(documents);

After this change:

// Azure CosmosDB path
const store = await AzureCosmosDBNoSQLVectorStore.fromDocuments([], embeddings, { credentials });
await store.delete({ filter: `SELECT * FROM c WHERE c.metadata.source = "${filename}"` });
await store.addDocuments(documents);

// FAISS path
const store = await FaissStore.load(faissStoreFolder, embeddings);
await removeDuplicateDocuments(store, filename);
await store.addDocuments(documents);

The solution is minimal and preserves all existing functionality while adding the requested deduplication capability.

Fixes #36.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• functionscdn.azureedge.net
  • Triggering command: node lib/install.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed.