diff --git a/.code-samples.meilisearch.yaml b/.code-samples.meilisearch.yaml index 87007b7a4..54b66addb 100644 --- a/.code-samples.meilisearch.yaml +++ b/.code-samples.meilisearch.yaml @@ -1533,3 +1533,19 @@ update_network_1: |- } } }' +search_parameter_reference_media_1: |- + curl \ + -X POST 'MEILISEARCH_URL/indexes/INDEX_NAME/search' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "hybrid": { + "embedder": "EMBEDDER_NAME" + }, + "media": { + "FIELD_A": "VALUE_A", + "FIELD_B" : { + "FIELD_C": "VALUE_B" + "FIELD_D": "VALUE_C" + } + } + }' diff --git a/learn/resources/experimental_features_overview.mdx b/learn/resources/experimental_features_overview.mdx index 90a16a8b1..1ec5ebc81 100644 --- a/learn/resources/experimental_features_overview.mdx +++ b/learn/resources/experimental_features_overview.mdx @@ -59,4 +59,6 @@ Activating or deactivating experimental features this way does not require you t | [Search query embedding cache](/learn/self_hosted/configure_meilisearch_at_launch#search-query-embedding-cache) | Enable a cache for search query embeddings | CLI flag or environment variable | | [Uncompressed snapshots](/learn/self_hosted/configure_meilisearch_at_launch#uncompressed-snapshots) | Disable snapshot compaction | CLI flag or environment variable | | [Maximum batch payload size](/learn/self_hosted/configure_meilisearch_at_launch#maximum-batch-payload-size) | Limit batch payload size | CLI flag or environment variable | -| [Disable new indexer](/learn/self_hosted/configure_meilisearch_at_launch) | Use previous settings indexer | CLI flag or environment variable | +| [Multimodal search](/reference/api/settings) | Enable multimodal search | API route | +| [Disable new indexer](/learn/self_hosted/configure_meilisearch_at_launch) | Use previous settings indexer | CLI flag or environment variable | + diff --git a/reference/api/experimental_features.mdx b/reference/api/experimental_features.mdx index 3393ceed9..b03cd95b6 100644 --- a/reference/api/experimental_features.mdx +++ b/reference/api/experimental_features.mdx @@ -26,7 +26,8 @@ The experimental API route is not compatible with all experimental features. Con "containsFilter": false, "editDocumentsByFunction": false, "network": false, - "chatCompletions": false + "chatCompletions": false, + "multimodal": false } ``` @@ -38,6 +39,7 @@ The experimental API route is not compatible with all experimental features. Con | **`editDocumentsByFunction`** | Boolean | `true` if feature is active, `false` otherwise | | **`network`** | Boolean | `true` if feature is active, `false` otherwise | | **`chatCompletions`** | Boolean | `true` if feature is active, `false` otherwise | +| **`multimodal`** | Boolean | `true` if feature is active, `false` otherwise | ## Get all experimental features @@ -58,7 +60,8 @@ Get a list of all experimental features that can be activated via the `/experime "containsFilter": false, "editDocumentsByFunction": false, "network": false, - "chatCompletions": false + "chatCompletions": false, + "multimodal": false } ``` @@ -87,6 +90,7 @@ Setting a field to `null` leaves its value unchanged. "containsFilter": false, "editDocumentsByFunction": false, "network": false, - "chatCompletions": false + "chatCompletions": false, + "multimodal": false } ``` diff --git a/reference/api/search.mdx b/reference/api/search.mdx index c5aeafa0b..4c84aeb61 100644 --- a/reference/api/search.mdx +++ b/reference/api/search.mdx @@ -97,6 +97,7 @@ By default, [this endpoint returns a maximum of 1000 results](/learn/resources/k | **[`vector`](#vector)** | Array of numbers | `null` | Search using a custom query vector | | **[`retrieveVectors`](#display-_vectors-in-response)** | Boolean | `false` | Return document vector data | | **[`locales`](#query-locales)** | Array of strings | `null` | Explicitly specify languages used in a query | +| **[`media`](#media)** | Object | `null` | Perform AI-powered search queries with multimodal content | ### Response @@ -1283,3 +1284,46 @@ For full control over the way Meilisearch detects languages during indexing and … } ``` + +### Media + +**Parameter**: `media`
+**Expected value**: Object
+**Default value**: `null` + + +This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it: + +```sh +curl \ + -X PATCH 'MEILISEARCH_URL/experimental-features/' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "multimodal": true + }' +``` + + +Specifies data to populate search fragments when performing multimodal searches. + +`media` must be an object whose fields must correspond to the data required by one [search fragment](/reference/api/settings#searchfragments). `media` must match a single search fragment. If `media` matches more than one fragment or no search fragments at all, Meilisearch will return an error. + +It is mandatory to specify an embedder when using `media`. `media` is incompatible with `vector`. + +#### Example + + + +```json +{ + "hits": [ + { + "id": 0, + "title": "DOCUMENT NAME", + … + } + … + ], + … +} +``` diff --git a/reference/api/settings.mdx b/reference/api/settings.mdx index ad7845256..a8873de4b 100644 --- a/reference/api/settings.mdx +++ b/reference/api/settings.mdx @@ -2512,7 +2512,9 @@ These embedder objects may contain the following fields: | **`binaryQuantized`** | Boolean | Empty | Once set to `true`, irreversibly converts all vector dimensions to 1-bit values | | **`indexingEmbedder`** | Object | Empty | Configures embedder to vectorize documents during indexing | | **`searchEmbedder`** | Object | Empty | Configures embedder to vectorize search queries | -| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders | +| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders | +| **`indexingFragments`** | Object | Empty | Configures multimodal embedding generation at indexing time | +| **`searchFragments`** | Object | Empty | Configures data handling during multimodal search | ### Get embedder settings @@ -2875,6 +2877,118 @@ Both fields must be an object and accept the same fields as a regular embedder, `indexingEmbedder` and `searchEmbedder` are incompatible with all other embedder sources. +##### `indexingFragments` + + +This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it: + +```sh +curl \ + -X PATCH 'MEILISEARCH_URL/experimental-features/' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "multimodal": true + }' +``` + + +`indexingFragments` specifies which fields in your documents should be used to generate multimodal embeddings. It must be an object with the following structure: + +```json + "FRAGMENT_NAME": { + "value": { + … + } + } +``` + +`FRAGMENT_NAME` can be any valid string. It must contain a single field, `value`. `value` must then follow your chosen model's specifications. + +For example, for [VoyageAI's multimodal embedding route](https://docs.voyageai.com/reference/multimodal-embeddings-api), `value` must be an object containing a `content` field. `content` itself must contain an array of objects with a `type` field. Depending on `type`'s value, you must include either `text`, `image_url`, or `image_base64`: + +```json +{ + "VOYAGE_FRAGMENT_NAME_A": { + "value": { + "content": [ + { + "type": "text", + "text": "A document called {{doc.title}} that can be described as {{doc.description}}" + } + ] + } + }, + "VOYAGE_FRAGMENT_NAME_B": { + "value": { + "content": [ + { + "type": "image_url", + "image_url": "{{doc.image_url}}" + } + ] + } + }, +} +``` + +Use Liquid templates to interpolate document data into the fragment fields, where `doc` gives you access to all fields within a document. + +`indexingFragments` is optional when using the `rest` source. + +`indexingFragments` is incompatible with all other embedder sources. + +Specifying a `documentTemplate` in an embedder using `indexingFragments` will result in an error. + +You must specify at least one valid fragment in `searchFragments` when using `indexingFragments`. + +##### `searchFragments` + + +This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it: + +```sh +curl \ + -X PATCH 'MEILISEARCH_URL/experimental-features/' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "multimodal": true + }' +``` + + +`searchFragments` instructs Meilisearch how to parse fields present in a query's [`media` search parameter](/reference/api/search#media). It must be an object following the same structure as the [`indexingFragments`](/reference/api/settings#indexingfragments) object: + +```json + "FRAGMENT_NAME": { + "value": { + … + } + } +``` + +As with `indexingFragments`, the content of `value` should follow your model's specification. + +Use Liquid templates to interpolate search query data into the fragment fields, where `media` gives you access to all multimodal data received with a query: + +```json +"SEARCH_FRAGMENT_A": { + "value": { + "content": [ + { + "type": "image_base64", + "image_base64": "data:{{media.image.mime}};base64,{{media.image.data}}" + } + ] + } +}, +``` + +`searchFragments` is optional when using the `rest` source. + +`searchFragments` is incompatible with all other embedder sources. + +You must specify at least one valid fragment in `indexingFragments` when using `searchFragments`. + #### Example diff --git a/reference/errors/error_codes.mdx b/reference/errors/error_codes.mdx index bf4c8dd1f..a77cea7a5 100644 --- a/reference/errors/error_codes.mdx +++ b/reference/errors/error_codes.mdx @@ -146,6 +146,14 @@ The given [`uid`](/reference/api/keys#uid) is invalid. The `uid` must follow the The value passed to [`attributesToSearchOn`](/reference/api/search#customize-attributes-to-search-on-at-search-time) is invalid. `attributesToSearchOn` accepts an array of strings indicating document attributes. Attributes given to `attributesToSearchOn` must be present in the [`searchableAttributes` list](/learn/relevancy/displayed_searchable_attributes#the-searchableattributes-list). +## `invalid_search_media` + +The value passed to [`media`](/reference/api/search#media) is not a valid JSON object. + +## `invalid_search_media_and_vector` + +The search query contains non-`null` values for both [`media`](/reference/api/search#media) and [`vector`](/reference/api/search#media). These two parameters are mutually exclusive, since `media` generates vector embeddings via the embedder configured in `hybrid`. + ## `invalid_content_type` The [Content-Type header](/reference/api/overview#content-type) is not supported by Meilisearch. Currently, Meilisearch only supports JSON, CSV, and NDJSON.