Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions mindsdb/integrations/handlers/valkey_handler/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Valkey Handler

This is the implementation of the Valkey handler for MindsDB, providing vector store capabilities using the [Valkey Search](https://valkey.io/) module.

## Prerequisites

- **Valkey Server** 9.0+ with the Search module enabled (e.g., `valkey/valkey-bundle` Docker image)
- **Python packages**: `valkey-glide>=2.4.0`, `numpy>=1.21.0`

## Installation

Install the handler dependencies:

```bash
pip install valkey-glide numpy
```

Or install via MindsDB extras:

```bash
pip install mindsdb[valkey]
```

## Connection

Create a Valkey vector store connection in MindsDB:

```sql
CREATE DATABASE my_valkey
WITH ENGINE = 'valkey',
PARAMETERS = {
"host": "localhost", -- Valkey server hostname (default: localhost)
"port": 6379, -- Valkey server port (default: 6379)
"password": "", -- Authentication password (optional)
"db": 0, -- Database number 0-15 (default: 0)
"vector_dimension": 384, -- Default embedding dimension (default: 384)
"distance_metric": "COSINE", -- COSINE, L2, or IP (default: COSINE)
"prefix": "doc:" -- Key prefix for document hashes (default: "doc:")
};
```

## Usage

### Create a Table (Vector Index)

```sql
CREATE TABLE my_valkey.my_collection
(SELECT * FROM my_model
WHERE content = 'sample text');
```

Or use the knowledge base pattern:

```sql
CREATE KNOWLEDGE BASE my_kb
USING
VECTOR STORE = my_valkey,
MODEL = my_embedding_model;
```

### Insert Data

```sql
INSERT INTO my_valkey.my_collection (id, content, embeddings, metadata)
VALUES ('doc1', 'Hello world', '[0.1, 0.2, ...]', '{"source": "web"}');
```

### Vector Similarity Search (KNN)

```sql
SELECT id, content, distance
FROM my_valkey.my_collection
WHERE search_vector = (SELECT embeddings FROM my_model WHERE content = 'query text')
LIMIT 5;
```

### Select by ID

```sql
SELECT id, content, metadata
FROM my_valkey.my_collection
WHERE id = 'doc1';
```

### Delete Documents

```sql
DELETE FROM my_valkey.my_collection
WHERE id = 'doc1';

DELETE FROM my_valkey.my_collection
WHERE id IN ('doc1', 'doc2', 'doc3');
```

### Drop Table (Index)

```sql
DROP TABLE my_valkey.my_collection;
```

## Configuration Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `host` | str | `localhost` | Valkey server hostname |
| `port` | int | `6379` | Valkey server port |
| `password` | str | `None` | Authentication password |
| `db` | int | `0` | Database number (0-15) |
| `vector_dimension` | int | `384` | Default embedding dimension for new indexes |
| `distance_metric` | str | `COSINE` | Distance metric: `COSINE`, `L2`, or `IP` |
| `prefix` | str | `doc:` | Key prefix for document hash keys |

## Distance Metrics

| Metric | Description | Use Case |
|--------|-------------|----------|
| `COSINE` | Cosine similarity (1 - cos_sim) | Text embeddings, normalized vectors |
| `L2` | Euclidean distance | Image embeddings, spatial data |
| `IP` | Inner product (negative dot product) | Pre-normalized vectors, recommendation |

## Architecture

The handler uses:
- **valkey-glide** (async client) with a synchronous wrapper (`asyncio.run_until_complete`)
- **HASH-based storage**: Documents are stored as Redis hashes with the key pattern `{prefix}{table}:{id}`
- **FT.CREATE**: Creates HNSW vector indexes with configurable dimensions and distance metrics
- **FT.SEARCH**: Executes KNN vector similarity queries with optional pre-filtering

## Running Tests

```bash
# Unit tests (no Valkey required)
pytest mindsdb/integrations/handlers/valkey_handler/tests/test_valkey_handler.py -v -k "Unit"

# Integration tests (requires running Valkey with Search module)
VALKEY_HOST=localhost VALKEY_PORT=6379 pytest mindsdb/integrations/handlers/valkey_handler/tests/test_valkey_handler.py -v -k "Integration"
```

### Running Valkey for Tests

```bash
docker run -d --name valkey-test -p 6379:6379 valkey/valkey-bundle:9.1
```
9 changes: 9 additions & 0 deletions mindsdb/integrations/handlers/valkey_handler/__about__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
__title__ = "MindsDB Valkey handler"
__package_name__ = "mindsdb_valkey_handler"
__version__ = "0.0.1"
__description__ = "MindsDB handler for Valkey Vector Store (via valkey-glide)"
__author__ = "Daria Korenieva"
__github__ = "https://github.com/mindsdb/mindsdb"
__pypi__ = "https://pypi.org/project/mindsdb/"
__license__ = "MIT"
__copyright__ = "Copyright 2026 - mindsdb"
33 changes: 33 additions & 0 deletions mindsdb/integrations/handlers/valkey_handler/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from mindsdb.integrations.libs.const import HANDLER_SUPPORT_LEVEL, HANDLER_TYPE

from .__about__ import __description__ as description
from .__about__ import __version__ as version
from .connection_args import connection_args, connection_args_example

try:
from .valkey_handler import ValkeyHandler as Handler

import_error = None
except Exception as e:
Handler = None
import_error = e

title = "Valkey"
name = "valkey"
type = HANDLER_TYPE.DATA
support_level = HANDLER_SUPPORT_LEVEL.COMMUNITY
icon_path = "icon.svg"

__all__ = [
"Handler",
"version",
"name",
"type",
"title",
"description",
"support_level",
"connection_args",
"connection_args_example",
"import_error",
"icon_path",
]
64 changes: 64 additions & 0 deletions mindsdb/integrations/handlers/valkey_handler/connection_args.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from collections import OrderedDict

from mindsdb.integrations.libs.const import HANDLER_CONNECTION_ARG_TYPE as ARG_TYPE

connection_args = OrderedDict(
host={
"type": ARG_TYPE.STR,
"description": "Valkey server hostname",
"required": False,
},
port={
"type": ARG_TYPE.INT,
"description": "Valkey server port",
"required": False,
},
password={
"type": ARG_TYPE.PWD,
"description": "Valkey authentication password",
"required": False,
"secret": True,
},
db={
"type": ARG_TYPE.INT,
"description": "Valkey database number (0-15)",
"required": False,
},
vector_dimension={
"type": ARG_TYPE.INT,
"description": "Default vector dimension for new indexes",
"required": False,
},
distance_metric={
"type": ARG_TYPE.STR,
"description": "Distance metric: COSINE, L2, or IP",
"required": False,
},
index_algorithm={
"type": ARG_TYPE.STR,
"description": "Vector index algorithm: HNSW (default) or FLAT",
"required": False,
},
prefix={
"type": ARG_TYPE.STR,
"description": "Key prefix for document hashes",
"required": False,
},
use_tls={
"type": ARG_TYPE.BOOL,
"description": "Enable TLS/SSL connection (required for AWS ElastiCache, MemoryDB)",
"required": False,
},
request_timeout={
"type": ARG_TYPE.INT,
"description": "Request timeout in milliseconds (default: 5000)",
"required": False,
},
)

connection_args_example = OrderedDict(
host="localhost",
port=6379,
vector_dimension=384,
distance_metric="COSINE",
)
1 change: 1 addition & 0 deletions mindsdb/integrations/handlers/valkey_handler/icon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions mindsdb/integrations/handlers/valkey_handler/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
valkey-glide>=2.4.0
numpy>=1.21.0,<3
Empty file.
Loading
Loading