You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/vectorchord/getting-started/overview.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,7 @@ docker run \
37
37
--name vectorchord-demo \
38
38
-e POSTGRES_PASSWORD=mysecretpassword \
39
39
-p 5432:5432 \
40
-
-d tensorchord/vchord-postgres:pg18-v0.5.3
40
+
-d tensorchord/vchord-postgres:pg18-v1.0.0
41
41
```
42
42
> In addition to the base image with the VectorChord extension, we provide an all-in-one image, `tensorchord/vchord-suite:pg18-latest`. This comprehensive image includes all official TensorChord extensions. Developers should select an image tag that is compatible with their extension's version, as indicated in [the support matrix](https://github.com/tensorchord/VectorChord-images?tab=readme-ov-file#support-matrix).
43
43
@@ -86,7 +86,7 @@ For more usage, please read:
86
86
-[Prewarm](/vectorchord/usage/prewarm)
87
87
-[Prefilter](/vectorchord/usage/prefilter)
88
88
-[Prefetch](/vectorchord/usage/prefetch)
89
-
-[Rerank In Table](/vectorchord/usage/rerank-in-table)
89
+
-[Rerank in Table](/vectorchord/usage/rerank-in-table)
Copy file name to clipboardExpand all lines: src/vectorchord/usage/indexing.md
+56-18Lines changed: 56 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -88,21 +88,23 @@ SET vchordrq.probes TO '10';
88
88
SELECT*FROM items ORDER BY embedding <=>'[3,1,2]'LIMIT10;
89
89
```
90
90
91
-
For large tables, the `build.internal` process costs huge time and memory. You can refer to [External Build](external-index-precomputation) to have a better experience.
92
-
93
-
For large tables, you may opt to use more shared memory to accelerate the process by setting `build.pin` to `true`.
91
+
For large tables, you may opt to use more shared memory to accelerate the process by setting `build.pin` to `2`.
94
92
95
93
```sql
96
94
CREATEINDEXON items USING vchordrq (embedding vector_l2_ops) WITH (options = $$
97
95
residual_quantization = true
98
-
build.pin=true
96
+
build.pin=2
99
97
[build.internal]
100
98
lists = [1000]
101
99
spherical_centroids = true
102
100
build_threads =8
103
101
$$);
104
102
```
105
103
104
+
For large tables, the `build.internal` process costs significant time and memory. Let `build.internal.kmeans_dimension` or the dimension be $D$, `build.internal.lists[-1]` be $C$, `build.internal.sampling_factor` be $F$, and `build.internal.build_threads` be $T$. The memory consumption is approximately $4CD(F + T + 1)$ bytes. You can moderately reduce these options for lower memory usage.
105
+
106
+
You can also refer to [External Build](external-index-precomputation) to offload the indexing workload to other machines.
@@ -140,21 +142,40 @@ The operator classes for `MaxSim` are available since version `0.3.0`.
140
142
141
143
#### `residual_quantization`
142
144
143
-
- Description: This index parameter determines whether residual quantization is used. If you not familiar with residual quantization, you can read this [blog](https://drscotthawley.github.io/blog/posts/2023-06-12-RVQ.html) for more information. In short, residual quantization is a technique that improves the accuracy of vector search by quantizing the residuals of the vectors.
145
+
- Description: This option determines whether residual quantization is used. If you are not familiar with residual quantization, you can read this [blog](https://drscotthawley.github.io/blog/posts/2023-06-12-RVQ.html) for more information. In short, residual quantization is a technique that improves the accuracy of vector search by quantizing the residuals of the vectors.
144
146
- Type: boolean
145
147
- Default: `false`
146
148
- Example:
147
149
-`residual_quantization = false` means that residual quantization is not used.
148
150
-`residual_quantization = true` means that residual quantization is used.
- Description: This option is a hint that specifies the degree of parallelism. In most cases, you do not need to change it. If you are using a CPU with more than `32` threads and wish to utilize more threads for PostgreSQL, you may set it to the number of threads for better performance.
155
+
- Type: integer
156
+
- Default: `32`
157
+
- Domain: `[1, 256]`
158
+
- Example:
159
+
-`degree_of_parallelism = 32` hints to the index that `32` or less processes may access on the index concurrently.
160
+
-`degree_of_parallelism = 64` hints to the index that `64` or less processes may access on the index concurrently.
- Description: This index parameter determines whether shared memory is used for indexing. For large tables, you can choose to enable this option to speed up the build process.
153
-
- Type: boolean
154
-
- Default: `false`
164
+
- Description: This option determines whether shared memory is used for indexing. For large tables, you can choose to enable this option to speed up the build process.
@@ -164,7 +185,7 @@ This is the default value of index building. The index will not be partitioned.
164
185
165
186
#### `build.internal.lists`
166
187
167
-
- Description: This index parameter determines the hierarchical structure of the vector space partitioning.
188
+
- Description: This option determines the hierarchical structure of the vector space partitioning.
168
189
- Type: list of integers
169
190
- Default:
170
191
-`[]` <badgetype="tip"text="since v0.3.0" />
@@ -173,11 +194,11 @@ This is the default value of index building. The index will not be partitioned.
173
194
-`build.internal.lists = []` means that the vector space is not partitioned.
174
195
-`build.internal.lists = [4096]` means the vector space is divided into $4096$ cells.
175
196
-`build.internal.lists = [4096, 262144]` means the vector space is divided into $4096$ cells, and those cells are further divided into $262144$ smaller cells.
176
-
- Note: The index partitions the vector space into multiple Voronoi cells based on centroids, iteratively creating a hierarchical space partition tree. Each leaf node in this tree represents a region with an associated list storing vectors in that region. During insertion, vectors are placed in lists corresponding to their appropriate leaf nodes. For queries, the index optimizes search by excluding lists whose leaf nodes are distant from the query vector, effectively pruning the search space. If the length of `lists` is $1$, the `lists` option should be no less than $4 * \sqrt{N}$, where $N$ is the number of vectors in the table.
197
+
- Note: The index partitions the vector space into multiple Voronoi cells based on centroids, iteratively creating a hierarchical space partition tree. Each leaf node in this tree represents a region with an associated list storing vectors in that region. During insertion, vectors are placed in lists corresponding to their appropriate leaf nodes. For queries, the index optimizes search by excluding lists whose leaf nodes are distant from the query vector, effectively pruning the search space. If the length of `lists` is $1$, the `lists` option should be no less than $4\sqrt{N}$, where $N$ is the number of vectors in the table.
177
198
178
199
#### `build.internal.spherical_centroids`
179
200
180
-
- Description: This index parameter determines whether to perform spherical K-means -- the centroids are L2 normalized after each iteration, you can refer to option `spherical` in [here](https://github.com/facebookresearch/faiss/wiki/Faiss-building-blocks:-clustering,-PCA,-quantization#additional-options).
201
+
- Description: This option determines whether to perform spherical K-means -- the centroids are L2 normalized after each iteration, you can refer to option `spherical` in [here](https://github.com/facebookresearch/faiss/wiki/Faiss-building-blocks:-clustering,-PCA,-quantization#additional-options).
181
202
- Type: boolean
182
203
- Default: `false`
183
204
- Example:
@@ -187,17 +208,17 @@ This is the default value of index building. The index will not be partitioned.
- Description: This index parameter determines the number of vectors the K-means algorithm samples per cluster. The higher this value, the slower the build, the greater the memory consumption in building, and the better search performance.
211
+
- Description: This option determines the number of vectors the K-means algorithm samples per cluster. The higher this value, the slower the build, the greater the memory consumption in building, and the better search performance.
191
212
- Type: integer
192
213
- Domain: `[0, 1024]`
193
214
- Default: `256`
194
215
- Example:
195
216
-`build.internal.sampling_factor = 256` means that the K-means algorithm samples $256 C$ vectors, where $C$ is the maximum value in `build.internal.lists`.
196
-
-`build.internal.sampling_factor = 1024` means that the K-means algorithm samples $1024 C$ vectors, where $C$ is the maximum value in `build.internal.lists`.
217
+
-`build.internal.sampling_factor = 32` means that the K-means algorithm samples $32 C$ vectors, where $C$ is the maximum value in `build.internal.lists`. This reduces K-means' time and memory usage to approximately $\frac{1}{8}$ of what it would be with the default value of `256`.
- Description: This index parameter determines the number of iterations for K-means algorithm. The higher this value, the slower the build.
221
+
- Description: This option determines the number of iterations for K-means algorithm. The higher this value, the slower the build.
201
222
- Type: integer
202
223
- Domain: `[0, 1024]`
203
224
- Default: `10`
@@ -207,14 +228,31 @@ This is the default value of index building. The index will not be partitioned.
207
228
208
229
#### `build.internal.build_threads`
209
230
210
-
- Description: This index parameter determines the number of threads used by K-means algorithm. The higher this value, the faster the build, and greater load on the server in building.
231
+
- Description: This option determines the number of threads used by K-means algorithm. The higher this value, the faster the build, and greater load on the server in building.
211
232
- Type: integer
212
233
- Domain: `[1, 255]`
213
234
- Default: `1`
214
235
- Example:
215
236
-`build.internal.build_threads = 1` means that the K-means algorithm uses $1$ thread.
216
237
-`build.internal.build_threads = 4` means that the K-means algorithm uses $4$ threads.
- Description: This option determines the K-means algorithm to be used.
242
+
- Type: object
243
+
- Example:
244
+
-`build.internal.kmeans_algorithm.lloyd = {}`. This uses Lloyd's algorithm. This is the default value.
245
+
-`build.internal.kmeans_algorithm.hierarchical = {}`. This uses hierarchical clustering. Compared to Lloyd's algorithm, this approach is much faster, but it may cause a loss of accuracy.
- Description: This option determines the dimension to use for K-means input and output. This feature employs dimensionality reduction and expansion via resampling, effectively reducing K-means' time and memory consumption, but it may cause a loss of accuracy.
250
+
- Type: union of integer and null
251
+
- Default: null
252
+
- Example:
253
+
- If this option is not set, this feature is disabled.
254
+
-`build.internal.kmeans_dimension = 100` means that K-means will process vectors with $100$ dimensions. For original vectors of $900$ dimensions, this reduces K-means' time and memory usage to approximately $\frac{1}{9}$ of what it would be without this feature.
@@ -250,7 +288,7 @@ This is the default value of index building. The index will not be partitioned.
250
288
-`SET vchordrq.epsilon = 0.1` indicates you are using a very optimistic lower bound estimation. You set it this way because your dataset is not sensitive to the lower bound estimation, for the precision you need.
251
289
-`SET vchordrq.epsilon = 4.0` indicates you are using a very pessimistic lower bound estimation. You set it this way because your dataset is not very sensitive to the lower bound estimation, for the precision you need.
252
290
253
-
#### `vchordrq.prewarm_dim` <badgetype="danger"text="deprecated in v0.4.0" />
291
+
#### `vchordrq.prewarm_dim` <badgetype="danger"text="removed in v0.4.0" />
254
292
255
293
- Description: The `vchordrq.prewarm_dim` GUC parameter is used to precompute the RaBitQ projection matrix for the specified dimensions. This can help to reduce the latency of the first query after the PostgreSQL cluster is started.
In the context of vector search, recall is the ratio of true nearest neighbors to approximate nearest neighbors returned by the index. For example, if the index retrieves $100$ approximate nearest neighbors and $97$ of them are true nearest neighbors, then the recall is $\frac{97}{100} = 0.97$.
0 commit comments