Memory Usage of xmen index

[phlobo]

Currently, the indexing procedure for SapBERT (embeddings + FAISS) is rather memory-intensive, as the complete FAISS index is constructed in memory. For large indices (e.g., Quaero) this can require > 32 GB of RAM.

A different RAM / accuracy tradeoff might be achieved in FAISS, but the impact on retrieval performance needs to be investigated.

[NargesFarrokhshad]

Hello @phlobo

As creating FAISS index is memory-intensive and time-consuming why don't you use another similarity searcher with faster searchers like SCANN?

[phlobo]

Thank you for your suggestion!

I don't think it's so much a problem of FAISS, but rather the current settings:
https://github.com/hpi-dhc/xmen/blob/main/xmen/linkers/faiss_indexer.py#L127

We will need to perform experiments of different settings and their effect on candidate retrieval performance.
This might could also mean that we replace FAISS (and maybe also NMSLib used by the TF-IDF linker) altogether.