Catalog dense vector

[christinach]

related to pulibrary/dacs_handbook#351

Adds a new handler to use it in the catalog semantic search. It works with defType lucene.

The dense vector field uses the knn algorithm https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#densevectorfield

https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-9.html#solr-9-0

The text embeddings were generated locally using pulibrary/dedup-text-embeddings#1 and the model 'multi-qa-mpnet-base-cos-v1'

To search using the knn index , solr expects a q param of "{!knn f=text_embeddings topK=10}[0.010338805615901947,0.02806314453482628,0.020983939990401268]" and with solr9 we need to set the defType to lucene.

http://localhost:61903/solr/orangelight-core-dev/select?defType=lucene&indent=true&q.op=OR&q=%7B!knn%20f%3Dtext_embeddings%20topK%3D10%7D%5B0.010338805615901947%2C0.02806314453482628%2C0.020983939990401268%5D&useParams=

{
  "responseHeader":{
    "status":0,
    "QTime":8,
    "params":{
      "q":"{!knn f=text_embeddings topK=10}[0.010338805615901947,0.02806314453482628,0.020983939990401268]",
      "defType":"lucene",
      "indent":"true",
      "q.op":"OR",
      "useParams":""
    }
  },
  "response":{
    "numFound":2,
    "start":0,
    "maxScore":0.9992631,
    "numFoundExact":true,
    "docs":[{
      "id":"9923849243506421",
      "title_display":"Vercingétorix. Histoire du pays gaulois depuis ses origines jusqu'à la conquête romaine",
      "author_display":["Colomb, Georges"],
      "score":0.9992631
    },{
      "id":"9923843813506421",
      "title_display":"The bright land",
      "author_display":["Fairbank, Janet Ayer"],
      "score":0.7314496
    }]
  },
  "facet_counts":{
    "facet_queries":{ },
    "facet_fields":{
      "format":[ ],
      "language_facet":[ ],
      "pub_date_start_sort":[ ],
      "advanced_location_s":[ ]
    },
    "facet_ranges":{ },
    "facet_intervals":{ },
    "facet_heatmaps":{ }
  }
}

[christinach]

needs defType lucene to work

[christinach]

vectorDimension= 3 is just for testing the knn query and the /semantic handler locally. It is not a realistic vector dimension.
The specific model by default creates vectors with dimensions 768.
In order to reduce the size of the vectors we would need to use a method like PCA which is also used in https://github.com/pulibrary/dedup-text-embeddings/blob/main/ipca_on_embeddings.py for a different purpose. Or train the model with a smaller size of vectors.