Feature: External Vector Accessor

[zhangjinpeng87]

Describe what you are looking for

I'm using usearch rust sdk to build a hybrid retrieval data store. I used usearch::Index to do semantic similarity search. In my case, I built a LSM-tree like architecture for the vector index part (so it introduced the capability of range filtering), latest put/delete happen in the latest usearch::Index, and as it grow, we flush it to disk as an segment (usearch::Index on disk).

In order to control the total memory usage of the whole vector index, I'm trying to just store u64 id and graph structure in the usearch::Index, and store the original vectors into a sequential file. The u64 id is the vector's offset in the sequential vector file. By separating the vectors from the Index, we can keep the Index part very small, and we mmap the vector file, so just these vectors blocks accessed is in RSS. so we need a way to access the original vector by id.

I don't use View because the sequential vector files can be used to do segment merging/compaction. Do you think such External Vector Accessor interface is valuable?

Can you contribute to the implementation?

• I can contribute

Is your feature request specific to a certain interface?

It applies to everything

Contact Details

zzzhangjinpeng@gmail.com

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[shankar-iyer]

Thanks for initiating this!

The exclude_vectors flag is there. But my understanding is that the flag has not been fully implemented. During serialization, the flag is checked and vectors are not serialized into the output file if flag is true. But on deserialize, there is no interface or lambda etc for the caller to provide the vectors from an external store. This comment right here :

USearch/include/usearch/index_dense.hpp

Line 1098 in 7306bb4

// We may not want to load the vectors from the same file, or allow attaching them afterwards

How about just adding 2 methods to index_dense_gt -

void allocate_vectors_lookup();
void set_vector_at_position(size_t position, f32_t const* vector, bool copy_vector = true);

Users call these 2 methods after calling load_from_stream(). We plan to do this in ClickHouse.

@ashvardanian