Incorporate ANN

[bamader]

Description

This PR introduces approximate neighbor search to our performance metrics notebook. It also creates a script that uses a relative grid-search to optimize the parameters for HNSW evaluation.

Related Issues

Closes #106

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.87%. Comparing base (977e2d6) to head (eaed1d7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #119   +/-   ##
=======================================
  Coverage   96.87%   96.87%           
=======================================
  Files           2        2           
  Lines          32       32           
=======================================
  Hits           31       31           
  Misses          1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[m-goggins]

Looks good overall! Left a couple of questions and small nits about documentation. I think the notes about the HNSW parameters could live in the scripts for now or in a big documentation file about our approach. Maybe just leave in the scripts for now and add the larger documentation later?

[bamader]

I also uploaded a few util scripts with the PR that might come in handy for one-off stuff in preparation for Azure (e.g., computing the HNSW index for a particular set of embeddings, or converting pickle-saved embedding Tensors into CPU format)