Skip to content

[MOD-10185] add SVS fp16 tests #740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Conversation

meiravgri
Copy link
Collaborator

@meiravgri meiravgri commented Aug 8, 2025

Overview

This PR introduces a comprehensive test suite for FP16 data types in the SVS algorithm.
The new test file test_svs_fp16.cpp is adapted from the existing test_svs.cpp to specifically validate SVS functionality with float16 vectors across single-index, multi-index, and tiered index configuration.

Key Changes

New Test File: tests/unit/test_svs_fp16.cpp

Test Class Structure

The file introduces three specialized test classes:

  1. FP16SVSTest: Core SVS functionality tests for single-index scenarios
  2. FP16SVSMultiTest: Multi-index SVS tests (inherits from FP16SVSTest)
  3. FP16SVSTieredIndexTest: Tiered index SVS tests (inherits from FP16SVSTest)
  • Type Definitions: runs float16 with quantization modes:

    • VecSimSvsQuant_NONE
    • VecSimSvsQuant_8
    • VecSimSvsQuant_8x8_LeanVec
  • Custom Vector Generation: Added GenerateVector() and GenerateAndAddVector() methods specifically for float16 with proper FP32→FP16 conversion

  • Data Conversion: All vector operations use vecsim_types::FP32_to_FP16() and vecsim_types::FP16_to_FP32() for proper type conversion

Test Coverage

Total Test Count: 50 tests (31 core + 10 multi-index + 9 tiered index)

Core SVS Tests (FP16SVSTest - 31 tests)

All essential SVS functionality tests adapted for FP16, including both adapted tests from the original suite and new FP16-specific tests:

Adapted from original test suite (26 tests):

  • Vector operations: svs_vector_add_test, svs_vector_update_test
  • Bulk operations: svs_bulk_vectors_add_delete_test
  • Index management: svs_indexing_same_vector, svs_reindexing_same_vector, svs_reindexing_same_vector_different_id
  • Batch iteration: svs_batch_iterator, svs_batch_iterator_non_unique_scores, svs_batch_iterator_reset, svs_batch_iterator_corner_cases
  • Index lifecycle: resizeIndex, svs_empty_index, test_delete_vector
  • Search algorithms: svs_vector_search_test_ip, svs_vector_search_test_l2, svs_vector_search_test_cosine
  • Performance optimization: preferAdHocOptimization, batchIteratorSwapIndices
  • Metrics and queries: rangeQuery, rangeQueryCosine
  • Debugging and info: test_svs_info, test_basic_svs_info_iterator, test_dynamic_svs_info_iterator
  • Size estimation: testSizeEstimation
  • Stress testing: sanity_reinsert_1280

New FP16-specific tests (5 tests):

  • svs_vector_search_test: Enhanced vector search test adapted for FP16
  • test_get_distance: Distance calculation validation for FP16
  • quant_modes: Comprehensive quantization mode validation
  • test_override_all: Vector override functionality testing
  • scalar_quantization_query: Scalar quantization query validation

Multi-Index Tests (FP16SVSMultiTest - 10 tests)

Specialized tests for multi-index scenarios where multiple vectors can share the same label:

  • vector_add_multiple_test: Adding multiple vectors with same label
  • vector_search_test: Search functionality in multi-index context
  • search_more_than_there_is: Edge case handling for search limits
  • find_better_score: Score optimization in multi-vector scenarios
  • find_better_score_after_pop: Score optimization after vector removal
  • reindexing_same_vector_different_id: Reindexing with different IDs
  • test_svs_info: Multi-index debug information validation
  • test_basic_svs_info_iterator: Multi-index debug iterator testing
  • rangeQuery: Range queries in multi-index context
  • svs_batch_iterator_basic: Batch iteration for multi-index

Tiered Index Tests (FP16SVSTieredIndexTest - 9 tests)

Tests for tiered index architecture with background indexing:

  • CreateIndexInstanceSingle/CreateIndexInstanceMulti: Index creation validation
  • RangeTestSingle/RangeTestMulti: Range query testing for both configurations
  • KNNSearch: K-nearest neighbor search in tiered context
  • deleteVector/deleteVectorMulti: Vector deletion in tiered indexes
  • BatchIteratorSingle/BatchIteratorMulti: Batch iteration for tiered indexes

Tests Excluded (17 tests)

Tests from the original test_svs.cpp that were intentionally not included, either because they are not relevant to For couldn't be reliably implemented with FP16 precision:

  • Timeout tests: testTimeoutReturn_topK, testTimeoutReturn_range, testTimeoutReturn_batch_iterator
  • Parameter validation: test_svs_parameter_combinations_and_defaults, test_svs_parameter_consistency_across_metrics
  • Runtime parameter resolution: resolve_ws_search_runtime_params, resolve_bc_search_runtime_params, resolve_use_search_history_runtime_params, resolve_epsilon_runtime_params
  • FP16 incompatible: svs_test_inf_score (infinity values not achievable with FP16)
  • Redundant/irrelevant: testInitialSizeEstimation (covered by testSizeEstimation), joinSearchParams (type not relevant), FitMemoryTest (no-op), logging_runtime_params

Additional Infrastructure Changes

SVS Index Implementation Enhancements

Enhanced Vector Data Access (src/VecSim/algorithms/svs/svs.h)

  • Implemented getStoredVectorDataByLabel(): Added comprehensive implementation for retrieving stored vector data by label (only supported for non-compressed indices, both single and multi-index configurations)

Query Result Debugging (src/VecSim/query_result_definitions.h)

  • Debug Print Operators: Added operator<< for VecSimQueryResult and VecSimQueryReply to enable better test debugging
  • Test-Only Features: Wrapped debug functionality in #ifdef BUILD_TESTS to keep production code clean
  • Enhanced Error Messages: Improved test failure diagnostics with detailed query result information (validateTopKSearchTest() function now prints all results of the assertion fails: ASSERT_TRUE(allUniqueResults(res)) << *res;)

Test Infrastructure Improvements

Enhanced Test Utilities (tests/unit/unit_test_utils.cpp & .h)

  • SVS Parameter Validation: Added validateSVSIndexAttributesInfo() function to validate SVS index parameters against svsInfoStruct values

Extended Test Data Generation (tests/utils/tests_utils.h)

  • FP16 Vector Generation: Added populate_float16_vec() function for generating random FP16 test vectors

@meiravgri meiravgri changed the title add fp16 tests [MOD-10185] add SVS fp16 tests Aug 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant