Leverage optimized native float32 vector scorers.

[ChrisHegarty]

Leverage optimized native float32 vector scorers.

The changes in this PR give approximately 2x performance increase for float32 vector operations across Linux/ Mac AArch64 and Linux x64 (both AVX2 and AVX 512).

The vector scorers leverage the native vector operations added by #130635.

The tests verify that the native scorers return similar values to that of the lucene scorers.

TODO: feature flag the new format so that we only create indices with that format when enabled.

[ChrisHegarty]

The micro benchmarks all show approx 2x performance improvement in scorer operations, all platforms. For example:

Apple Mac M2, AArch64
Low-level benchmark results. Compare dotProductLuceneWithCopy to dotProductNativeWithNativeSeg, bigger is better.

Benchmark                                                (size)  Mode  Cnt    Score    Error  Units
JDKVectorFloat32Benchmark.dotProductLucene                 1024  avgt   15   60.448 ±  4.160  ns/op
JDKVectorFloat32Benchmark.dotProductLuceneWithCopy         1024  avgt   15  115.741 ± 11.562  ns/op
JDKVectorFloat32Benchmark.dotProductNativeWithHeapSeg      1024  avgt   15   60.691 ±  4.329  ns/op
JDKVectorFloat32Benchmark.dotProductNativeWithNativeSeg    1024  avgt   15   59.111 ±  0.751  ns/op

Scorer benchmark. Compare dotProductLuceneQuery to dotProductNativeQuery, bigger is better.

Benchmark                                     (dims)   Mode  Cnt  Score   Error   Units
Float32ScorerBenchmark.dotProductLucene         1024  thrpt    5  3.522 ± 0.025  ops/us
Float32ScorerBenchmark.dotProductLuceneQuery    1024  thrpt    5  3.969 ± 0.110  ops/us
Float32ScorerBenchmark.dotProductNative         1024  thrpt    5  7.772 ± 0.060  ops/us
Float32ScorerBenchmark.dotProductNativeQuery    1024  thrpt    5  8.260 ± 0.123  ops/us
Float32ScorerBenchmark.dotProductScalar         1024  thrpt    5  0.602 ± 0.003  ops/us

[ldematte]

I've concentrated mainly on the native (C) part and its interaction (NativeAccess/VectorLibrary), and that looks good to me.
I have looked at the benchmarks and tests and they look sensible, but I think it's better to have another pair of eyes on the other parts (VectorScorer/lucene interaction)

[ldematte]

Benchmarks looks good, and they are a testament of the goodness of Panama and its usage in Lucene! Hopefully OpenJDK will get rid of the bug "soon" :)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[ChrisHegarty]

I'm considering whether or not it is worth back porting this to 8.19.x.

[benwtrent]

IMO, this is way too complex to actually backport to the 8.19 branch.

However, "backporting" the change to allow the scorers to be used on indices created in the 8.19 series (which is the majority of indices still), is a very nice thing :)

[ChrisHegarty]

Yeah. Oddly, because we already have a ES813FlatVectorFormat, it means that we can just hook the native scorer implementation in and quite a bunch of existing flat indices will get it. For HNSW, we just use the Lucene99 format so we cannot as easily do that.

[benwtrent]

@ChrisHegarty @ldematte where are we on this PR? Do we want to do a chunk at a time and maybe by-pass the new format just now?

FYI, I am seeing other places where we should do something like this and we can hook in off-heap scoring like this for rescoring vectors as well. #131048

[ldematte]

My 2c: I would do one step at a time, and merge this PR as-is (if it is correct -- e.g. if CI passes). The part that I'm familiar with (the NativeAccess stuff) lgtm.

[ChrisHegarty]

I deliberately paused this PR, in order to switch focus onto the bulk scoring API in Lucene. I think that we should reflow this API and the native scorers on top of the upcoming Lucene bulk scoring API. That way we can use prefetch instructions on the next vector as we're scoring the current vector - or just score say 4 vectors per loop itr. It will also reduce the cost of Java to native transitions - which are small, but a little more costly on x64 since we use a shared segment for the index.

[benwtrent]

I deliberately paused this PR, in order to switch focus onto the bulk scoring API in Lucene. I think that we should reflow this API and the native scorers on top of the upcoming Lucene bulk scoring API.

I agree, but you mean that no native code should be done with single function calls?

I would expect this PR and an additional one would be needed. Where the follow up would be turning on bulk-scoring (e.g. 4 vectors at a time, or something).