Adjust openai_vector benchmark for gpu indexing

[mayya-sharipova]

• Adds GPU acceleration support via the use_gpu_indexing parameter
• Replaces the merge-wait polling with an explicit force-merge operation
• Optimizes indexing performance by disabling merge throttling and refresh intervals

---

Draft PR for now as "index.vectors.indexing.use_gpu" is currently only enabled under Feature flag.

For GPU indexing pass the following params.json file to a rally run:

params.json

{
  "use_gpu_indexing" : true, 
  "vector_index_type" : "hnsw",   
  "initial_indexing_bulk_indexing_clients" : 8 
}

Notes:

./rally race ...--runtime-jdk=25 --track-params=<.../params.json> --on-error=abort

---

Notes

• use_gpu_indexing –  supports true to enforce gpu indexing or false to enforce CPU indexing
• vector_index_type : supports hnsw or int8_hnsw for GPU indexing
• initial_indexing_bulk_indexing_clients – we recommend to set indexing clients = number of CPU cores/2.
• runtime-jdk should be at least 22
• LD_LIBRARY_PATH should be defined and set in rally.ini

From 9.3, there is a new option for using base64 encoded vectors for indexing

• use base64 encoded vectors for much faster indexing. For example, for openai_vector track, add the following params in params.json:

 "initial_indexing_corpora" : "openai_initial_indexing_base64",  
 "parallel_indexing_corpora": "openai_parallel_indexing_base64"

vim ~/.rally/rally.ini

...
[system]
env.name = local
passenv = PATH,LD_LIBRARY_PATH

---

Example running openai_vector track:

params_gpu.json

{
  "use_gpu_indexing" : true,
  "vector_index_type" :  "int8_hnsw",
  "mapping_type" : "vectors-only-with-docid",
  "initial_indexing_bulk_indexing_clients" : 8
}

params_cpu.json

{
  "use_gpu_indexing" : false,
  "vector_index_type" : "int8_hnsw",
  "mapping_type" : "vectors-only-with-docid",
  "initial_indexing_bulk_indexing_clients" : 8
}

./rally race --track-path /home/ubuntu/elastic/rally-tracks/openai_vector \
--pipeline from-sources \
--revision current \
--car "16gheap" \
--user-tags "tag:openai_vector_gpu" \
--report-file /home/ubuntu/elastic/benchmarks/rally_results/openai_vector_gpu.md \
--runtime-jdk=25   \
--track-params=/home/ubuntu/elastic/benchmarks/rally/params.json \
--on-error=abort

---

Troubleshoot

• Check ~/.rally/benchmarks/races/ /rally-node-0/logs/server/rally-benchmark.log. If you see similar output in the elasticsearch server logs:
  [o.e.x.g.GPUSupport ] [elasticsearch-0] Found compatible GPU [NVIDIA L4] (id: [0])

it means your setup is successful, and Elasticsearch can use GPU. If you don’t see this line from GPUSupport, you should see an explanation why elasticsearch can’t use your GPU device.

[mayya-sharipova]

Closing this PR, as it is not needed anymore.
Updates instructions for v9.3 can be found here.