Skip to content

Commit 9fc49d6

Browse files
guimachiavellidureuill
guimachiavelli
and
dureuill
authored
v1.16: Multimodal search (#3312)
--------- Co-authored-by: Louis Dureuil <[email protected]>
1 parent 98fbe2c commit 9fc49d6

File tree

6 files changed

+193
-5
lines changed

6 files changed

+193
-5
lines changed

‎.code-samples.meilisearch.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1545,3 +1545,19 @@ update_network_1: |-
15451545
}
15461546
}
15471547
}'
1548+
search_parameter_reference_media_1: |-
1549+
curl \
1550+
-X POST 'MEILISEARCH_URL/indexes/INDEX_NAME/search' \
1551+
-H 'Content-Type: application/json' \
1552+
--data-binary '{
1553+
"hybrid": {
1554+
"embedder": "EMBEDDER_NAME"
1555+
},
1556+
"media": {
1557+
"FIELD_A": "VALUE_A",
1558+
"FIELD_B" : {
1559+
"FIELD_C": "VALUE_B"
1560+
"FIELD_D": "VALUE_C"
1561+
}
1562+
}
1563+
}'

‎learn/resources/experimental_features_overview.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,4 +59,6 @@ Activating or deactivating experimental features this way does not require you t
5959
| [Search query embedding cache](/learn/self_hosted/configure_meilisearch_at_launch#search-query-embedding-cache) | Enable a cache for search query embeddings | CLI flag or environment variable |
6060
| [Uncompressed snapshots](/learn/self_hosted/configure_meilisearch_at_launch#uncompressed-snapshots) | Disable snapshot compaction | CLI flag or environment variable |
6161
| [Maximum batch payload size](/learn/self_hosted/configure_meilisearch_at_launch#maximum-batch-payload-size) | Limit batch payload size | CLI flag or environment variable |
62-
| [Disable new indexer](/learn/self_hosted/configure_meilisearch_at_launch) | Use previous settings indexer | CLI flag or environment variable |
62+
| [Multimodal search](/reference/api/settings) | Enable multimodal search | API route |
63+
| [Disable new indexer](/learn/self_hosted/configure_meilisearch_at_launch) | Use previous settings indexer | CLI flag or environment variable |
64+

‎reference/api/experimental_features.mdx

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ The experimental API route is not compatible with all experimental features. Con
2626
"containsFilter": false,
2727
"editDocumentsByFunction": false,
2828
"network": false,
29-
"chatCompletions": false
29+
"chatCompletions": false,
30+
"multimodal": false
3031
}
3132
```
3233

@@ -38,6 +39,7 @@ The experimental API route is not compatible with all experimental features. Con
3839
| **`editDocumentsByFunction`** | Boolean | `true` if feature is active, `false` otherwise |
3940
| **`network`** | Boolean | `true` if feature is active, `false` otherwise |
4041
| **`chatCompletions`** | Boolean | `true` if feature is active, `false` otherwise |
42+
| **`multimodal`** | Boolean | `true` if feature is active, `false` otherwise |
4143

4244
## Get all experimental features
4345

@@ -58,7 +60,8 @@ Get a list of all experimental features that can be activated via the `/experime
5860
"containsFilter": false,
5961
"editDocumentsByFunction": false,
6062
"network": false,
61-
"chatCompletions": false
63+
"chatCompletions": false,
64+
"multimodal": false
6265
}
6366
```
6467

@@ -87,6 +90,7 @@ Setting a field to `null` leaves its value unchanged.
8790
"containsFilter": false,
8891
"editDocumentsByFunction": false,
8992
"network": false,
90-
"chatCompletions": false
93+
"chatCompletions": false,
94+
"multimodal": false
9195
}
9296
```

‎reference/api/search.mdx

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ By default, [this endpoint returns a maximum of 1000 results](/learn/resources/k
9797
| **[`vector`](#vector)** | Array of numbers | `null` | Search using a custom query vector |
9898
| **[`retrieveVectors`](#display-_vectors-in-response)** | Boolean | `false` | Return document vector data |
9999
| **[`locales`](#query-locales)** | Array of strings | `null` | Explicitly specify languages used in a query |
100+
| **[`media`](#media)** | Object | `null` | Perform AI-powered search queries with multimodal content |
100101

101102
### Response
102103

@@ -1285,3 +1286,46 @@ For full control over the way Meilisearch detects languages during indexing and
12851286
…
12861287
}
12871288
```
1289+
1290+
### Media <NoticeTag type="experimental" label="experimental" />
1291+
1292+
**Parameter**: `media`<br />
1293+
**Expected value**: Object<br />
1294+
**Default value**: `null`
1295+
1296+
<Note>
1297+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
1298+
1299+
```sh
1300+
curl \
1301+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
1302+
-H 'Content-Type: application/json' \
1303+
--data-binary '{
1304+
"multimodal": true
1305+
}'
1306+
```
1307+
</Note>
1308+
1309+
Specifies data to populate search fragments when performing multimodal searches.
1310+
1311+
`media` must be an object whose fields must correspond to the data required by one [search fragment](/reference/api/settings#searchfragments). `media` must match a single search fragment. If `media` matches more than one fragment or no search fragments at all, Meilisearch will return an error.
1312+
1313+
It is mandatory to specify an embedder when using `media`. `media` is incompatible with `vector`.
1314+
1315+
#### Example
1316+
1317+
<CodeSamplesSearchParameterReferenceMedia1 />
1318+
1319+
```json
1320+
{
1321+
"hits": [
1322+
{
1323+
"id": 0,
1324+
"title": "DOCUMENT NAME",
1325+
…
1326+
}
1327+
…
1328+
],
1329+
…
1330+
}
1331+
```

‎reference/api/settings.mdx

Lines changed: 115 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2512,7 +2512,9 @@ These embedder objects may contain the following fields:
25122512
| **`binaryQuantized`** | Boolean | Empty | Once set to `true`, irreversibly converts all vector dimensions to 1-bit values |
25132513
| **`indexingEmbedder`** | Object | Empty | Configures embedder to vectorize documents during indexing |
25142514
| **`searchEmbedder`** | Object | Empty | Configures embedder to vectorize search queries |
2515-
| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders |
2515+
| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders |
2516+
| **`indexingFragments`** | Object | Empty | Configures multimodal embedding generation at indexing time |
2517+
| **`searchFragments`** | Object | Empty | Configures data handling during multimodal search |
25162518

25172519
### Get embedder settings
25182520

@@ -2875,6 +2877,118 @@ Both fields must be an object and accept the same fields as a regular embedder,
28752877

28762878
`indexingEmbedder` and `searchEmbedder` are incompatible with all other embedder sources.
28772879

2880+
##### `indexingFragments` <NoticeTag type="experimental" label="experimental" />
2881+
2882+
<Note>
2883+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
2884+
2885+
```sh
2886+
curl \
2887+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
2888+
-H 'Content-Type: application/json' \
2889+
--data-binary '{
2890+
"multimodal": true
2891+
}'
2892+
```
2893+
</Note>
2894+
2895+
`indexingFragments` specifies which fields in your documents should be used to generate multimodal embeddings. It must be an object with the following structure:
2896+
2897+
```json
2898+
"FRAGMENT_NAME": {
2899+
"value": {
2900+
…
2901+
}
2902+
}
2903+
```
2904+
2905+
`FRAGMENT_NAME` can be any valid string. It must contain a single field, `value`. `value` must then follow your chosen model's specifications.
2906+
2907+
For example, for [VoyageAI's multimodal embedding route](https://docs.voyageai.com/reference/multimodal-embeddings-api), `value` must be an object containing a `content` field. `content` itself must contain an array of objects with a `type` field. Depending on `type`'s value, you must include either `text`, `image_url`, or `image_base64`:
2908+
2909+
```json
2910+
{
2911+
"VOYAGE_FRAGMENT_NAME_A": {
2912+
"value": {
2913+
"content": [
2914+
{
2915+
"type": "text",
2916+
"text": "A document called {{doc.title}} that can be described as {{doc.description}}"
2917+
}
2918+
]
2919+
}
2920+
},
2921+
"VOYAGE_FRAGMENT_NAME_B": {
2922+
"value": {
2923+
"content": [
2924+
{
2925+
"type": "image_url",
2926+
"image_url": "{{doc.image_url}}"
2927+
}
2928+
]
2929+
}
2930+
},
2931+
}
2932+
```
2933+
2934+
Use Liquid templates to interpolate document data into the fragment fields, where `doc` gives you access to all fields within a document.
2935+
2936+
`indexingFragments` is optional when using the `rest` source.
2937+
2938+
`indexingFragments` is incompatible with all other embedder sources.
2939+
2940+
Specifying a `documentTemplate` in an embedder using `indexingFragments` will result in an error.
2941+
2942+
You must specify at least one valid fragment in `searchFragments` when using `indexingFragments`.
2943+
2944+
##### `searchFragments` <NoticeTag type="experimental" label="experimental" />
2945+
2946+
<Note>
2947+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
2948+
2949+
```sh
2950+
curl \
2951+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
2952+
-H 'Content-Type: application/json' \
2953+
--data-binary '{
2954+
"multimodal": true
2955+
}'
2956+
```
2957+
</Note>
2958+
2959+
`searchFragments` instructs Meilisearch how to parse fields present in a query's [`media` search parameter](/reference/api/search#media). It must be an object following the same structure as the [`indexingFragments`](/reference/api/settings#indexingfragments) object:
2960+
2961+
```json
2962+
"FRAGMENT_NAME": {
2963+
"value": {
2964+
…
2965+
}
2966+
}
2967+
```
2968+
2969+
As with `indexingFragments`, the content of `value` should follow your model's specification.
2970+
2971+
Use Liquid templates to interpolate search query data into the fragment fields, where `media` gives you access to all multimodal data received with a query:
2972+
2973+
```json
2974+
"SEARCH_FRAGMENT_A": {
2975+
"value": {
2976+
"content": [
2977+
{
2978+
"type": "image_base64",
2979+
"image_base64": "data:{{media.image.mime}};base64,{{media.image.data}}"
2980+
}
2981+
]
2982+
}
2983+
},
2984+
```
2985+
2986+
`searchFragments` is optional when using the `rest` source.
2987+
2988+
`searchFragments` is incompatible with all other embedder sources.
2989+
2990+
You must specify at least one valid fragment in `indexingFragments` when using `searchFragments`.
2991+
28782992
#### Example
28792993

28802994
<CodeSamplesUpdateEmbedders1 />

‎reference/errors/error_codes.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,14 @@ The given [`uid`](/reference/api/keys#uid) is invalid. The `uid` must follow the
146146

147147
The value passed to [`attributesToSearchOn`](/reference/api/search#customize-attributes-to-search-on-at-search-time) is invalid. `attributesToSearchOn` accepts an array of strings indicating document attributes. Attributes given to `attributesToSearchOn` must be present in the [`searchableAttributes` list](/learn/relevancy/displayed_searchable_attributes#the-searchableattributes-list).
148148

149+
## `invalid_search_media`
150+
151+
The value passed to [`media`](/reference/api/search#media) is not a valid JSON object.
152+
153+
## `invalid_search_media_and_vector`
154+
155+
The search query contains non-`null` values for both [`media`](/reference/api/search#media) and [`vector`](/reference/api/search#media). These two parameters are mutually exclusive, since `media` generates vector embeddings via the embedder configured in `hybrid`.
156+
149157
## `invalid_content_type`
150158

151159
The [Content-Type header](/reference/api/overview#content-type) is not supported by Meilisearch. Currently, Meilisearch only supports JSON, CSV, and NDJSON.

0 commit comments

Comments
 (0)