Merge pull request #2159 from Danielle9897/RDoc-3468-extend-querying-pre-made-embeddings

RDoc-3468 Get embeddings generation tasks details + exception info

Author: reebhub

docs/ai-integration/generating-embeddings/content/_overview-csharp.mdx

@@ -0,0 +1,184 @@
+import Admonition from '@theme/Admonition';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+<Admonition type="note" title="">
+
+* RavenDB can serve as a vector database, see [Why choose RavenDB as your vector database](../../../ai-integration/vector-search/ravendb-as-vector-database.mdx#why-choose-ravendb-as-your-vector-database).
+
+* Vector search can be performed on:   
+   * Raw text stored in your documents.   
+   * Pre-made embeddings that you created yourself and stored using these [Data types](../../../ai-integration/vector-search/data-types-for-vector-search.mdx#numerical-data).  
+   * Pre-made embeddings that are automatically generated from your document content by RavenDB's  
+     **embeddings generation tasks** using external service providers, as explained below.
+* In this article:
+  * [Embeddings generation - overview](../../../ai-integration/generating-embeddings/overview.mdx#embeddings-generation---overview)
+     * [Embeddings generation - process flow](../../../ai-integration/generating-embeddings/overview.mdx#embeddings-generation---process-flow)
+     * [Supported providers](../../../ai-integration/generating-embeddings/overview.mdx#supported-providers)
+  * [Creating an embeddings generation task](../../../ai-integration/generating-embeddings/overview.mdx#creating-an-embeddings-generation-task)
+  * [Monitoring the tasks](../../../ai-integration/generating-embeddings/overview.mdx#monitoring-the-tasks)
+  * [Get embeddings generation task details](../../../ai-integration/generating-embeddings/overview.mdx#get-embeddings-generation-task-details)  
+
+</Admonition>
+
+## Embeddings generation - overview
+
+<Admonition type="note" title="">
+
+#### Embeddings generation - process flow
+    
+* **Define an Embeddings Generation Task**:  
+  Specify a [connection string](../../../ai-integration/connection-strings/connection-strings-overview.mdx) that defines the AI provider and model for generating embeddings.  
+  Define the source content - what parts of the documents will be used to create the embeddings.  
+
+* **Source content is processed**:  
+  1. The task extracts the specified content from the documents.  
+  2. If a processing script is defined, it transforms the content before further processing.  
+  3. The text is split according to the defined chunking method; a separate embedding will be created for each chunk.  
+  4. Before contacting the provider, RavenDB checks the [embeddings cache](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection)
+     to determine whether an embedding already exists for the given content from that provider.
+  5. If a matching embedding is found, it is reused, avoiding unnecessary requests.  
+     If no cached embedding is found, the transformed and chunked content is sent to the configured AI provider.  
+
+* **Embeddings are generated by the AI provider**:  
+  The provider generates embeddings and sends them back to RavenDB.  
+  If quantization was defined in the task, RavenDB applies it to the embeddings before storing them.
+
+* **Embeddings are stored in your database**:  
+  * Each embedding is stored as an attachment in a [dedicated collection](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-collection).  
+  * RavenDB maintains an [embeddings cache](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection),
+    allowing reuse of embeddings for the same source content and reducing provider calls.
+    Cached embeddings expire after a configurable duration.
+
+* **Perform vector search:**  
+  Once the embeddings are stored, you can perform vector searches on your document content by:  
+  * Running a [dynamic query](../../../ai-integration/vector-search/vector-search-using-dynamic-query.mdx#querying-pre-made-embeddings-generated-by-tasks), which automatically creates an auto-index for the search.  
+  * Defining a [static index](../../../ai-integration/vector-search/vector-search-using-static-index.mdx#indexing-pre-made-text-embeddings) to store and query embeddings efficiently.  
+
+      The query search term is split into chunks, and each chunk is looked up in the cache.  
+      If not found, RavenDB requests an embedding from the provider and caches it.  
+      The embedding (cached or newly created) is then used to compare against stored vectors. 
+
+* **Continuous processing**:  
+  * Embeddings generation tasks are [Ongoing Tasks](../../../studio/database/tasks/ongoing-tasks/general-info.mdx) that process documents as they change.  
+    Before contacting the provider after a document change, the task first checks the cache to see if a matching embedding already exists, avoiding unnecessary requests.
+  * The requests to generate embeddings from the source text are sent to the provider in batches.  
+    The batch size is configurable, see the [Ai.Embeddings.MaxBatchSize](../../../server/configuration/ai-integration-configuration.mdx#aiembeddingsmaxbatchsize) configuration key.  
+  * A failed embeddings generation task will retry after the duration set in the  
+    [Ai.Embeddings.MaxFallbackTimeInSec](../../../server/configuration/ai-integration-configuration.mdx#aiembeddingsmaxfallbacktimeinsec) configuration key.
+
+</Admonition>
+
+<Admonition type="note" title="">
+
+#### Supported providers
+    
+* The following service providers are supported for auto-generating embeddings using tasks:
+
+    * [OpenAI & OpenAI-compatible providers](../../../ai-integration/connection-strings/open-ai.mdx)
+    * [Azure Open AI](../../../ai-integration/connection-strings/azure-open-ai.mdx)
+    * [Google AI](../../../ai-integration/connection-strings/google-ai.mdx)
+    * [Hugging Face](../../../ai-integration/connection-strings/hugging-face.mdx)
+    * [Ollama](../../../ai-integration/connection-strings/ollama.mdx)
+    * [Mistral AI](../../../ai-integration/connection-strings/mistral-ai.mdx)
+    * [bge-micro-v2](../../../ai-integration/connection-strings/embedded.mdx) (a local embedded model within RavenDB)
+
+</Admonition>
+
+![flow chart](../assets/embeddings-generation-task-flow.png)
+
+![flow chart](../assets/vector-search-flow.png)
+
+## Creating an embeddings generation task
+
+* An embeddings generation tasks can be created from:
+    * The **AI Tasks view in the Studio**, where you can create, edit, and delete tasks. Learn more in [AI Tasks - list view](../../../ai-integration/ai-tasks-list-view.mdx).
+    * The **Client API** - see [Configuring an embeddings generation task - from the Client API](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-client-api)
+* From the Studio:  
+
+     ![Add ai task 1](../assets/add-ai-task-1.png)
+
+     1. Go to the **AI Hub** menu.
+     2. Open the **AI Tasks** view.
+     3. Click **Add AI Task** to add a new task.
+
+     ![Add ai task 2](../assets/add-ai-task-2.png)
+
+* See the complete details of the task configuration in the [Embeddings generation task](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx) article.
+
+## Monitoring the tasks
+
+* The status and state of each embeddings generation task are visible in the [AI Tasks - list view](../../../ai-integration/ai-tasks-list-view.mdx).
+
+* Task performance and activity over time can be analyzed in the _AI Tasks Stats_ view,  
+  where you can track processing duration, batch sizes, and overall progress.  
+  Learn more about the functionality of the stats view in the [Ongoing Tasks Stats](../../../studio/database/stats/ongoing-tasks-stats/overview.mdx) article.
+
+* The number of embeddings generation tasks across all databases can also be monitored using [SNMP](../../../server/administration/snmp/snmp-overview.mdx).  
+  The following SNMP OIDs provide relevant metrics:
+  * [5.1.11.25](../../../server/administration/snmp/snmp-overview.mdx#511125) – Total number of enabled embeddings generation tasks.
+  * [5.1.11.26](../../../server/administration/snmp/snmp-overview.mdx#511126) – Total number of active embeddings generation tasks.
+
+## Get embeddings generation task details
+
+* Besides viewing the list of tasks in the [AI Tasks - list view](../../../ai-integration/ai-tasks-list-view.mdx) in the Studio,  
+  you can also retrieve embeddings generation task details programmatically.
+
+* This is useful when issuing a vector search query that references an embeddings generation task,  
+  where it's important to verify that the task exists beforehand. For example:  
+  * when [Querying pre-made embeddings generated by tasks](../../../ai-integration/vector-search/vector-search-using-dynamic-query#querying-pre-made-embeddings-generated-by-tasks)
+  * or when [Indexing numerical data and querying using text input](../../../ai-integration/vector-search/vector-search-using-static-index#indexing-numerical-data-and-querying-using-text-input)
+
+* There are two ways to check if an embeddings generation task exists:
+  * Using `GetOngoingTaskInfoOperation`.  
+  * Accessing the full list of embeddings generation tasks from the database record.
+
+
+<Tabs groupId='languageSyntax'>
+<TabItem value="Get_task_info_via_operataion" label="Get_task_info_via_operataion">
+```csharp
+// Define the get task operation, pass the task NAME
+var getOngoingTaskOp = 
+    new GetOngoingTaskInfoOperation("theEmbeddingsGenerationTaskName", OngoingTaskType.EmbeddingsGeneration);
+    
+// Execute the operation by by passing it to Maintenance.Send
+// Explicitly cast the result to the "EmbeddingsGeneration" type 
+var task = (EmbeddingsGeneration)store.Maintenance.Send(getOngoingTaskOp);
+    
+// Verify the task exists
+if (task != null)
+{
+    // Access any of the task details
+    var taskStatus = task.TaskState;
+    
+    // Access the task identifier
+    var taskIdentifier = task.Configuration.Identifier;
+}
+```
+</TabItem>
+<TabItem value="Get_task_info_via_database_record" label="Get_task_info_via_database_record">
+```csharp
+// Define the get database record operation, pass your database name
+var getDatabaseRecordOp = new GetDatabaseRecordOperation("yourDatabaseName");
+  
+// Execute the operation by passing it to Maintenance.Send
+var dbRecord = store.Maintenance.Server.Send(getDatabaseRecordOp);
+  
+// Access the list of embeddings generation tasks
+var tasks = dbRecord.EmbeddingsGenerations;
+    
+if (tasks.Count > 0)
+{
+    // Access the first task
+    var task = tasks[0];
+      
+    // Access any of the task details
+    var isTaskDisabled = task.Disabled;
+      
+    // Access the task identifier
+    var taskIdentifier = task.Identifier;
+}
+```
+</TabItem>
+</Tabs>