[ENH]: SPANN - Delete empty PLs

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • We now delete a head and its associated PL once all of its points are outdated after reassign. Previously, we'd only do it during garbage collection.
  • This should reduce the size of hnsw index, reduce peak memory usage of compactor, and make compactions faster.
  • The same_head logic during splitting is also adjusted to account for negative distances that can happen for cosine and IP
• New functionality
  • ...

Test plan

How are these changes tested?
Ran the local repro that used to cause a stack overflow. Now it does so with very low prob. Previously it would stack overflow 100% of the time

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

None

Observability plan

None

Documentation Changes

None

[sanketkedia]

• [ENH]: SPANN - Delete empty PLs #5882 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

SPANN: Eager Deletion of Empty Posting Lists & Improved Split Logic

This PR refactors parts of SpannIndexWriter to immediately remove a head and its associated posting list (PL) once all points in the PL become outdated, rather than waiting for periodic garbage-collection. A new helper (try_delete_posting_list) centralises the deletion logic, is invoked from split, merge, and reassignment paths, and enforces a safe deletion order (first from HNSW, then from the PL). Additionally, the distance comparison used to detect an unchanged centroid during splits now takes the absolute value to handle negative distances returned by cosine/IP metrics. Several code paths were hardened for concurrent deletion scenarios and improved error propagation.

Key Changes

• Added async helper try_delete_posting_list() with full HNSW + PL cleanup and concurrency guard via posting_list_partitioned_mutex.
• Invoked try_delete_posting_list() after: split reassignment (collect_and_reassign_split_points), neighbour reassignment (collect_and_reassign_nearby_points), and cluster merge paths.
• Reworked error handling when fetching a PL: early‐exit on Ok(None) and explicit error propagation instead of silent ignore.
• Distance check in split logic now uses .distance(...).abs() < 1e-6 to tolerate negative distances for cosine / inner-product spaces.
• Consistent deletion order enforced: remove head from HNSW before deleting its PL to avoid dangling references.
• Misc stats counters and tracing retained; no API surface changes.

Affected Areas

• rust/index/src/spann/types.rs – methods dealing with splits, merges, reassign, GC, and new helper

This summary was automatically generated by @propel-code-bot

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
chroma-index/spann::types::tests::test_reassign	View Logs