From ccc22eba15d425d31b31c327339dfac9a6ce6483 Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Wed, 20 Aug 2025 12:00:35 +0300 Subject: [PATCH 1/6] Added Active-Active documentation page --- docs/active_active.rst | 223 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 223 insertions(+) create mode 100644 docs/active_active.rst diff --git a/docs/active_active.rst b/docs/active_active.rst new file mode 100644 index 0000000000..33b990930c --- /dev/null +++ b/docs/active_active.rst @@ -0,0 +1,223 @@ +Active-Active +============= + +MultiDBClient explanation +-------------------------- + +Starting from redis-py 6.5.0 we introduce a new type of client to communicate +with databases in Active-Active setup. `MultiDBClient` is a wrapper around multiple +Redis or Redis Cluster clients, each of them has 1:1 relation to specific +database. `MultiDBClient` in most of the cases provides the same API as any other +client for the best user experience. + +The core feature of `MultiDBClient` is automaticaly triggered failover depends on the +database healthiness. The pre-condition is that each database that is configured +to be used by MultiDBClient are eventually consistent, so client could choose +any database in any point of time for communication. `MultiDBClient` always communicates +with single database, so there's 1 active and N passive databases that acting as a +stand-by replica. By default, active database is choosed based on the weights that +has to be assigned for each database. + +We have two mechanisms to verify database healthiness: `Healthcheck` and +`Failure Detector`. + +The very basic configuration you need to setup a `MultiDBClient`: + +.. code:: python + + // Expected active database (highest weight) + database1_config = DatabaseConfig( + weight=1.0, + from_url="redis://host1:port1", + client_kwargs={ + 'username': "username", + 'password': "password", + } + ) + + // Passive database (stand-by replica) + database2_config = DatabaseConfig( + weight=0.9, + from_url="redis://host2:port2", + client_kwargs={ + 'username': "username", + 'password': "password", + } + ) + + config = MultiDbConfig( + databases_config=[database1_config, database2_config], + ) + + client = MultiDBClient(config) + + +Healthcheck +----------- + +By default, we're using healthcheck based on `ECHO` command to verify that database is +reachable and ready to serve requests (`PING` guarantees first, but not the second). +Additionaly, you can add your own healthcheck implementation and extend a list of +healthecks + +All healthchecks are running in the background with given interval and configuration +defined in `MultiDBConfig` class. + + +Failure Detector +---------------- + +Unlike healthcheck, `Failure Detector` verifies database healthiness based on organic +trafic, so the default one reacts to any command failures within a sliding window of +seconds and mark database as unhealthy if threshold has been exceeded. You can extend +a list of failure detectors providing your own implementation, configuration defined +in `MultiDBConfig` class. + + +Databases configuration +----------------------- + +You have to provide a configuration for each database in setup separately, using +`DatabaseConfig` class per database. As mentioned, there's an undelying instance +of `Redis` or `RedisCluster` client for each database, so you can pass all the +arguments related to them via `client_kwargs` argument. + +.. code:: python + + database_config = DatabaseConfig( + weight=1.0, + client_kwargs={ + 'host': 'localhost', + 'port': 6379, + 'username': "username", + 'password': "password", + } + ) + +It also supports `from_url` or `from_pool` capabilites to setup a client using +Redis URL or custom `ConnectionPool` object. + +.. code:: python + + database_config1 = DatabaseConfig( + weight=1.0, + from_url="redis://host1:port1", + client_kwargs={ + 'username': "username", + 'password': "password", + } + ) + + database_config2 = DatabaseConfig( + weight=0.9, + from_pool=connection_pool, + ) + +The only exception from `client_kwargs` is the retry configuration. We do not allow +to pass underlying `Retry` object to avoid nesting retries. All the retries are +controlled by top-level `Retry` object that you can setup via `command_retry` +argument (check `MultiDBConfig`) + + +Pipeline +-------- + +`MultiDBClient` supports pipeline mode with guaranteed pipeline retry in case +of failover. Unlike, the `Redis` and `RedisCluster` clients you cannot +execute transactions via pipeline mode, only via `transaction` method +on `MultiDBClient`. This was done for better retries handling in case +of failover. + +The overall interface for pipeline execution is the same, you can +pipeline commands using chaining calls or context manager. + +.. code:: python + + // Chaining + client = MultiDBClient(config) + pipe = client.pipeline() + pipe.set('key1', 'value1') + pipe.get('key1') + pipe.execute() // ['OK', 'value1'] + + // Context manager + client = MultiDBClient(config) + with client.pipeline() as pipe: + pipe.set('key1', 'value1') + pipe.get('key1') + pipe.execute() // ['OK', 'value1'] + + +Transaction +----------- + +`MultiDBClient` supports transaction execution via `transaction()` method +with guaranteed transaction retry in case of failover. Like any other +client it accepts a callback with underlying `Pipeline` object to build +your transaction for atomic execution + +CAS behaviour supported as well, so you can provide a list of keys to track. + +.. code:: python + + client = MultiDBClient(config) + + def callback(pipe: Pipeline): + pipe.set('key1', 'value1') + pipe.get('key1') + + client.transaction(callback, 'key1') // ['OK1', 'value1'] + + +Pub/Sub +------- + +`MultiDBClient` supports Pub/Sub mode with guaranteed re-subscription +to the same channels in case of failover. So the expectation is that +both publisher and subscriber are using `MultiDBClient` instance to +provide seamless experience in terms of failover. + +1. Subscriber failover to another database and re-subscribe to the same +channels. + +2. Publisher failover to another database and starts publishing +messages to the same channels. + +However, it's still possible to lose messages if order of failover +will be reversed. + +Like the other clients, there's two main methods to consume messages: +in the main thread and in the separate thread + +.. code:: python + + client = MultiDBClient(config) + p = client.pubsub() + + // In the main thread + while True: + message = p.get_message() + if message: + // do something with the message + time.sleep(0.001) + + +.. code:: python + + // In separate thread + client = MultiDBClient(config) + p = client.pubsub() + messages_count = 0 + data = json.dumps({'message': 'test'}) + + def handler(message): + nonlocal messages_count + messages_count += 1 + + // Assign a handler and run in a separate thread. + p.ssubscribe(**{'test-channel': handler}) + pubsub_thread = pubsub.run_in_thread(sleep_time=0.1, daemon=True) + + for _ in range(10): + client.publish('test-channel', data) + sleep(0.1) From 8e49c9a5fad48ef7d9a838e277de753b2d4244e3 Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Tue, 26 Aug 2025 15:15:30 +0300 Subject: [PATCH 2/6] Added documentation for Active-Active --- docs/active_active.rst | 162 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 161 insertions(+), 1 deletion(-) diff --git a/docs/active_active.rst b/docs/active_active.rst index 33b990930c..47b7f6cea1 100644 --- a/docs/active_active.rst +++ b/docs/active_active.rst @@ -21,6 +21,12 @@ has to be assigned for each database. We have two mechanisms to verify database healthiness: `Healthcheck` and `Failure Detector`. +To be able to use `MultiDBClient` you need to install a `pybreaker` package: + +.. code:: python + + pip install pybreaker>=1.4.0 + The very basic configuration you need to setup a `MultiDBClient`: .. code:: python @@ -63,6 +69,71 @@ healthecks All healthchecks are running in the background with given interval and configuration defined in `MultiDBConfig` class. +Lag-Aware Healthcheck +~~~~~~~~~~~~~~~~~~~~~ + +This is a special type of healthcheck available for Redis Software and Redis Cloud +that utilizes REST API endpoint to obtain an information about synchronisation lag +between given database and all other databases in Active-Active setup. + +To be able to use this type of healthcheck, first you need to adjust your +`DatabaseConfig` to expose `health_check_url` used by your deployment. +By default, your Cluster FQDN should be used as URL, unless you have +some kind of reverse proxy behind an actual REST API endpoint. + +.. code:: python + + database1_config = DatabaseConfig( + weight=1.0, + from_url="redis://host1:port1", + health_check_url="https://c1.deployment-name-000000.cto.redislabs.com" + client_kwargs={ + 'username': "username", + 'password': "password", + } + ) + +Since, Lag-Aware Healthcheck only available for Redis Software and Redis Cloud +it's not in the list of the default healthchecks for `MultiDBClient`. You have +to provide it manually during client configuration or in runtime. + +.. code:: python + + // Configuration option + config = MultiDbConfig( + databases_config=[database1_config, database2_config], + health_checks=[ + LagAwareHealthCheck(auth_basic=('username','password'), verify_tls=False) + ] + ) + + client = MultiDBClient(config) + +.. code:: python + + // In runtime + client = MultiDBClient(config) + client.add_health_check( + LagAwareHealthCheck(auth_basic=('username','password'), verify_tls=False) + ) + +As mentioned we utilise REST API endpoint for Lag-Aware healthchecks, so it accepts +different type of HTTP-related configuration: authentication credentials, request +timeout, TLS related configuration, etc. (check `LagAwareHealthCheck` class). + +You can also specify `lag_aware_tolerance` parameter to specify the tolerance in MS +of lag between databases that your application could tolerate. + +.. code:: python + + LagAwareHealthCheck( + rest_api_port=9443, + auth_basic=('username','password'), + lag_aware_tolerance=150, + verify_tls=True, + ca_file="path/to/file" + ) + Failure Detector ---------------- @@ -215,9 +286,98 @@ in the main thread and in the separate thread messages_count += 1 // Assign a handler and run in a separate thread. - p.ssubscribe(**{'test-channel': handler}) + p.subscribe(**{'test-channel': handler}) pubsub_thread = pubsub.run_in_thread(sleep_time=0.1, daemon=True) for _ in range(10): client.publish('test-channel', data) sleep(0.1) + + +OSS Cluster API support +----------------------- + +As mentioned `MultiDBClient` also supports integration with OSS Cluster API +databases. If you're instantiating client using Redis URL, the only change +you need comparing to standalone client is the `client_class` argument. +DNS server will resolve given URL and will point you to one of the node that +could be used to discover overall cluster topology. + +.. code:: python + + config = MultiDbConfig( + client_class=RedisCluster, + databases_config=[database1_config, database2_config], + ) + +If you would like to specify the exact node to use for topology +discovery, you can specify it the same way `RedisCluster` does + +.. code:: python + + // Expected active database (highest weight) + database1_config = DatabaseConfig( + weight=1.0, + client_kwargs={ + 'username': "username", + 'password': "password", + 'startup_nodes': [ClusterNode('host1', 'port1')], + } + ) + + // Passive database (stand-by replica) + database2_config = DatabaseConfig( + weight=0.9, + client_kwargs={ + 'username': "username", + 'password': "password", + 'startup_nodes': [ClusterNode('host2', 'port2')], + } + ) + + config = MultiDbConfig( + client_class=RedisCluster, + databases_config=[database1_config, database2_config], + ) + +Sharded Pub/Sub +~~~~~~~~~~~~~~~ + +If you would like to use a Sharded Pub/Sub capabilities make sure to use +correct Pub/Sub configuration. + +.. code:: python + + client = MultiDBClient(config) + p = client.pubsub() + + // In the main thread + while True: + // Reads messaage from sharded channels. + message = p.get_sharded_message() + if message: + // do something with the message + time.sleep(0.001) + + +.. code:: python + + // In separate thread + client = MultiDBClient(config) + p = client.pubsub() + messages_count = 0 + data = json.dumps({'message': 'test'}) + + def handler(message): + nonlocal messages_count + messages_count += 1 + + // Assign a handler and run in a separate thread. + p.ssubscribe(**{'test-channel': handler}) + + // Proactively executes get_sharded_pubsub() method + pubsub_thread = pubsub.run_in_thread(sleep_time=0.1, daemon=True, sharded_pubsub=True) + + for _ in range(10): + client.spublish('test-channel', data) + sleep(0.1) \ No newline at end of file From d83a1eaab87320020a1190603a1384e4100c9e68 Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Mon, 1 Sep 2025 11:07:06 +0300 Subject: [PATCH 3/6] Refactored docs --- .../{active_active.rst => multi_database.rst} | 145 +++++++++++------- 1 file changed, 89 insertions(+), 56 deletions(-) rename docs/{active_active.rst => multi_database.rst} (66%) diff --git a/docs/active_active.rst b/docs/multi_database.rst similarity index 66% rename from docs/active_active.rst rename to docs/multi_database.rst index 47b7f6cea1..92145c3223 100644 --- a/docs/active_active.rst +++ b/docs/multi_database.rst @@ -1,22 +1,23 @@ -Active-Active -============= +Multi-Database Management +========================= MultiDBClient explanation -------------------------- -Starting from redis-py 6.5.0 we introduce a new type of client to communicate -with databases in Active-Active setup. `MultiDBClient` is a wrapper around multiple -Redis or Redis Cluster clients, each of them has 1:1 relation to specific -database. `MultiDBClient` in most of the cases provides the same API as any other -client for the best user experience. - -The core feature of `MultiDBClient` is automaticaly triggered failover depends on the -database healthiness. The pre-condition is that each database that is configured -to be used by MultiDBClient are eventually consistent, so client could choose -any database in any point of time for communication. `MultiDBClient` always communicates -with single database, so there's 1 active and N passive databases that acting as a -stand-by replica. By default, active database is choosed based on the weights that -has to be assigned for each database. +The `MultiDBClient` (introduced in version 6.5.0) manages connections to multiple +Redis databases and provides automatic failover when one database becomes unavailable. +Think of it as a smart load balancer that automatically switches to a healthy database +when your primary one goes down, ensuring your application stays online. +`MultiDBClient` in most of the cases provides the same API as any other client for +the best user experience. + +The core feature of MultiDBClient is its ability to automatically trigger failover +when an active database becomes unhealthy.The pre-condition is that all databases +that are configured to be used by `MultiDBClient` are eventually consistent, so client +could choose any database in any point in time for communication. `MultiDBClient` +always communicates with single database, so there's 1 active and N passive +databases that are acting as a stand-by replica. By default, active database is +chosen based on the weights that have to be assigned for each database. We have two mechanisms to verify database healthiness: `Healthcheck` and `Failure Detector`. @@ -58,42 +59,70 @@ The very basic configuration you need to setup a `MultiDBClient`: client = MultiDBClient(config) -Healthcheck ------------ +Health Monitoring +----------------- +The `MultiDBClient` uses two complementary mechanisms to ensure database availability: + +Health Checks (Proactive Monitoring) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -By default, we're using healthcheck based on `ECHO` command to verify that database is -reachable and ready to serve requests (`PING` guarantees first, but not the second). -Additionaly, you can add your own healthcheck implementation and extend a list of -healthecks +These checks run continuously in the background at configured intervals to proactively +detect database issues. They run in the background with a given interval and +configuration defined in the `MultiDBConfig` class. -All healthchecks are running in the background with given interval and configuration -defined in `MultiDBConfig` class. +By default, MultiDBClient sends ECHO commands to verify each database is healthy. -Lag-Aware Healthcheck +**Custom Health Checks** +~~~~~~~~~~~~~~~~~~~~~ +You can add custom health checks for specific requirements: + +.. code:: python + + from redis.multidb.healthcheck import AbstractHealthCheck + from redis.retry import Retry + from redis.utils import dummy_fail + + + class PingHealthCheck(AbstractHealthCheck): + def __init__(self, retry: Retry): + super().__init__(retry=retry) + + def check_health(self, database) -> bool: + return self._retry.call_with_retry( + lambda: self._returns_pong(database), + lambda _: dummy_fail() + ) + + def _returns_pong(self, database) -> bool: + expected_message = ["PONG", b"PONG"] + actual_message = database.client.execute_command("PING") + return actual_message in expected_message + +**Lag-Aware Healthcheck (Redis Enterprise Only)** ~~~~~~~~~~~~~~~~~~~~~ This is a special type of healthcheck available for Redis Software and Redis Cloud -that utilizes REST API endpoint to obtain an information about synchronisation lag -between given database and all other databases in Active-Active setup. +that utilizes a REST API endpoint to obtain information about the synchronisation +lag between a given database and all other databases in an Active-Active setup. -To be able to use this type of healthcheck, first you need to adjust your -`DatabaseConfig` to expose `health_check_url` used by your deployment. -By default, your Cluster FQDN should be used as URL, unless you have -some kind of reverse proxy behind an actual REST API endpoint. +To use this healthcheck, first you need to adjust your `DatabaseConfig` +to expose `health_check_url` used by your deployment. By default, your +Cluster FQDN should be used as URL, unless you have some kind of +reverse proxy behind an actual REST API endpoint. .. code:: python database1_config = DatabaseConfig( weight=1.0, from_url="redis://host1:port1", - health_check_url="https://c1.deployment-name-000000.cto.redislabs.com" + health_check_url="https://c1.deployment-name-000000.project.env.com" client_kwargs={ 'username': "username", 'password': "password", } ) -Since, Lag-Aware Healthcheck only available for Redis Software and Redis Cloud +Since, Lag-Aware Healthcheck is only available for Redis Software and Redis Cloud it's not in the list of the default healthchecks for `MultiDBClient`. You have to provide it manually during client configuration or in runtime. @@ -135,23 +164,26 @@ of lag between databases that your application could tolerate. ) -Failure Detector ----------------- +Failure Detection (Reactive Monitoring) +~~~~~~~~~~~~~~~~~~~~~ -Unlike healthcheck, `Failure Detector` verifies database healthiness based on organic -trafic, so the default one reacts to any command failures within a sliding window of -seconds and mark database as unhealthy if threshold has been exceeded. You can extend -a list of failure detectors providing your own implementation, configuration defined -in `MultiDBConfig` class. +The failure detector watches actual command failures and marks databases as unhealthy +when error rates exceed thresholds within a sliding time window of a few seconds. +This catches issues that proactive health checks might miss during real traffic. +You can extend the list of failure detectors by providing your own implementation, +configuration defined in the `MultiDBConfig` class. Databases configuration ----------------------- -You have to provide a configuration for each database in setup separately, using -`DatabaseConfig` class per database. As mentioned, there's an undelying instance -of `Redis` or `RedisCluster` client for each database, so you can pass all the -arguments related to them via `client_kwargs` argument. +Each database needs a `DatabaseConfig` that specifies how to connect. + +Method 1: Using client_kwargs (most flexible) +~~~~~~~~~~~~~~~~~~~~~ + +There's an underlying instance of `Redis` or `RedisCluster` client for each database, +so you can pass all the arguments related to them via `client_kwargs` argument: .. code:: python @@ -165,11 +197,10 @@ arguments related to them via `client_kwargs` argument. } ) -It also supports `from_url` or `from_pool` capabilites to setup a client using -Redis URL or custom `ConnectionPool` object. - -.. code:: python +Method 2: Using Redis URL +~~~~~~~~~~~~~~~~~~~~~~~~~ +```python database_config1 = DatabaseConfig( weight=1.0, from_url="redis://host1:port1", @@ -179,15 +210,17 @@ Redis URL or custom `ConnectionPool` object. } ) - database_config2 = DatabaseConfig( - weight=0.9, - from_pool=connection_pool, - ) +Method 3: Using Custom Connection Pool +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +```python + database_config2 = DatabaseConfig( + weight=0.9, + from_pool=connection_pool, + ) -The only exception from `client_kwargs` is the retry configuration. We do not allow -to pass underlying `Retry` object to avoid nesting retries. All the retries are -controlled by top-level `Retry` object that you can setup via `command_retry` -argument (check `MultiDBConfig`) +**Important**: Don't pass `Retry` objects in `client_kwargs`. `MultiDBClient` +handles all retries at the top level through the `command_retry` configuration. Pipeline @@ -300,7 +333,7 @@ OSS Cluster API support As mentioned `MultiDBClient` also supports integration with OSS Cluster API databases. If you're instantiating client using Redis URL, the only change you need comparing to standalone client is the `client_class` argument. -DNS server will resolve given URL and will point you to one of the node that +DNS server will resolve given URL and will point you to one of the nodes that could be used to discover overall cluster topology. .. code:: python From 2ab08b2128802d68eda68d210145bdf9e3538c84 Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Mon, 1 Sep 2025 11:54:19 +0300 Subject: [PATCH 4/6] Refactored pipeline and transaction section --- docs/multi_database.rst | 69 +++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 30 deletions(-) diff --git a/docs/multi_database.rst b/docs/multi_database.rst index 92145c3223..eb80703831 100644 --- a/docs/multi_database.rst +++ b/docs/multi_database.rst @@ -200,7 +200,8 @@ so you can pass all the arguments related to them via `client_kwargs` argument: Method 2: Using Redis URL ~~~~~~~~~~~~~~~~~~~~~~~~~ -```python +.. code:: python + database_config1 = DatabaseConfig( weight=1.0, from_url="redis://host1:port1", @@ -213,7 +214,8 @@ Method 2: Using Redis URL Method 3: Using Custom Connection Pool ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -```python +.. code:: python + database_config2 = DatabaseConfig( weight=0.9, from_pool=connection_pool, @@ -223,28 +225,32 @@ Method 3: Using Custom Connection Pool handles all retries at the top level through the `command_retry` configuration. -Pipeline --------- +Pipeline Operations +------------------- + +The `MultiDBClient` supports pipeline mode with guaranteed retry functionality during +failover scenarios. Unlike standard `Redis` and `RedisCluster` clients, transactions +cannot be executed through pipeline mode - use the dedicated `transaction()` method +instead. This design choice ensures better retry handling during failover events. -`MultiDBClient` supports pipeline mode with guaranteed pipeline retry in case -of failover. Unlike, the `Redis` and `RedisCluster` clients you cannot -execute transactions via pipeline mode, only via `transaction` method -on `MultiDBClient`. This was done for better retries handling in case -of failover. +Pipeline operations support both chaining calls and context manager patterns: -The overall interface for pipeline execution is the same, you can -pipeline commands using chaining calls or context manager. +Chaining approach +~~~~~~~~~~~~~~~~~ .. code:: python - // Chaining client = MultiDBClient(config) pipe = client.pipeline() pipe.set('key1', 'value1') pipe.get('key1') pipe.execute() // ['OK', 'value1'] - // Context manager +Context Manager Approach +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: python + client = MultiDBClient(config) with client.pipeline() as pipe: pipe.set('key1', 'value1') @@ -255,12 +261,13 @@ pipeline commands using chaining calls or context manager. Transaction ----------- -`MultiDBClient` supports transaction execution via `transaction()` method -with guaranteed transaction retry in case of failover. Like any other -client it accepts a callback with underlying `Pipeline` object to build -your transaction for atomic execution +The `MultiDBClient` provides transaction support through the `transaction()` +method with guaranteed retry capabilities during failover. Like other +`Redis` clients, it accepts a callback function that receives a `Pipeline` +object for building atomic operations. -CAS behaviour supported as well, so you can provide a list of keys to track. +CAS behavior is fully supported by providing a list of +keys to monitor: .. code:: python @@ -276,22 +283,21 @@ CAS behaviour supported as well, so you can provide a list of keys to track. Pub/Sub ------- -`MultiDBClient` supports Pub/Sub mode with guaranteed re-subscription -to the same channels in case of failover. So the expectation is that -both publisher and subscriber are using `MultiDBClient` instance to -provide seamless experience in terms of failover. +The MultiDBClient offers Pub/Sub functionality with automatic re-subscription +to channels during failover events. For optimal failover handling, +both publishers and subscribers should use MultiDBClient instances. -1. Subscriber failover to another database and re-subscribe to the same -channels. +1. **Subscriber failover**: Automatically reconnects to an alternative database +and re-subscribes to the same channels -2. Publisher failover to another database and starts publishing -messages to the same channels. +2. **Publisher failover**: Seamlessly switches to an alternative database and +continues publishing to the same channels -However, it's still possible to lose messages if order of failover -will be reversed. +**Note**: Message loss may occur if failover events happen in reverse order +(publisher fails before subscriber). -Like the other clients, there's two main methods to consume messages: -in the main thread and in the separate thread +Main Thread Message Processing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: python @@ -306,6 +312,9 @@ in the main thread and in the separate thread time.sleep(0.001) +Background Thread Processing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + .. code:: python // In separate thread From 881580064c2513e4581753327e4dddc71365a4f0 Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Wed, 17 Sep 2025 12:00:45 +0300 Subject: [PATCH 5/6] Updated docs --- .github/wordlist.txt | 10 +++++++ docs/multi_database.rst | 63 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 70 insertions(+), 3 deletions(-) diff --git a/.github/wordlist.txt b/.github/wordlist.txt index 150f96a624..48ea9e8737 100644 --- a/.github/wordlist.txt +++ b/.github/wordlist.txt @@ -1,6 +1,7 @@ APM ARGV BFCommands +balancer CacheImpl CAS CFCommands @@ -10,10 +11,17 @@ ClusterNodes ClusterPipeline ClusterPubSub ConnectionPool +config CoreCommands +DatabaseConfig +DNS EVAL EVALSHA +failover +FQDN Grokzen's +Healthcheck +healthchecks INCR IOError Instrumentations @@ -21,6 +29,7 @@ JSONCommands Jaeger Ludovico Magnocavallo +MultiDBClient McCurdy NOSCRIPT NUMPAT @@ -52,6 +61,7 @@ SpanKind Specfiying StatusCode TCP +TLS TOPKCommands TimeSeriesCommands Uptrace diff --git a/docs/multi_database.rst b/docs/multi_database.rst index eb80703831..abca4493d4 100644 --- a/docs/multi_database.rst +++ b/docs/multi_database.rst @@ -61,6 +61,13 @@ The very basic configuration you need to setup a `MultiDBClient`: Health Monitoring ----------------- +To avoid false positives, you can configure amount of health check probes and also +define one of the health check policies to evaluate probes result. + +**HealthCheckPolicies.HEALTHY_ALL** - (default) All probes should be successful +**HealthCheckPolicies.HEALTHY_MAJORITY** - Majority of probes should be successful +**HealthCheckPolicies.HEALTHY_ANY** - Any of probes should be successful + The `MultiDBClient` uses two complementary mechanisms to ensure database availability: Health Checks (Proactive Monitoring) @@ -102,7 +109,7 @@ You can add custom health checks for specific requirements: ~~~~~~~~~~~~~~~~~~~~~ This is a special type of healthcheck available for Redis Software and Redis Cloud -that utilizes a REST API endpoint to obtain information about the synchronisation +that utilizes a REST API endpoint to obtain information about the synchronization lag between a given database and all other databases in an Active-Active setup. To use this healthcheck, first you need to adjust your `DatabaseConfig` @@ -146,7 +153,7 @@ to provide it manually during client configuration or in runtime. LagAwareHealthCheck(auth_basic=('username','password'), verify_tls=False) ) -As mentioned we utilise REST API endpoint for Lag-Aware healthchecks, so it accepts +As mentioned we utilize REST API endpoint for Lag-Aware healthchecks, so it accepts different type of HTTP-related configuration: authentication credentials, request timeout, TLS related configuration, etc. (check `LagAwareHealthCheck` class). @@ -174,6 +181,40 @@ You can extend the list of failure detectors by providing your own implementatio configuration defined in the `MultiDBConfig` class. +Failover strategy +~~~~~~~~~~~~~~~~~ + +This component is responsible for failover when active database becomes unavailable. +By default, we're using `WeightBasedFailoverStrategy` to pick a database with the +highest weight to failover. You can provide your own strategy if you would like +to have your custom mechanism of failover. + +.. code:: python + + class CustomFailoverStrategy(FailoverStrategy): + def __init__(self): + self._databases: Databases = None + + def database(self) -> SyncDatabase: + for database, _ in self._databases: + random_int = random.randint(0, 1) + + if random_int == 1 and database.circuit.state == State.CLOSED: + return database + + // Exception should be raised if theres no suitable databases for failover + raise NoValidDatabaseException("No available database for failover") + +In case if there's no available databases for failover, we raise `TemporaryUnavailableException`. +This exception signals that you can still trying to send requests until final +`NoValidDatabaseException` will be thrown. The window for requests is configurable +and depends on two parameters `failover_attempts` and `failover_delay`. By default, +`failover_attempts=10` and `failover_delay=12s`, which means that you can still send requests +for 10*12 = 120 seconds until final exception will be thrown. In meanwhile, you can switch to +another data source (cache) and if healthy database will apears you can switch back making +this transparent to the end user. + + Databases configuration ----------------------- @@ -422,4 +463,20 @@ correct Pub/Sub configuration. for _ in range(10): client.spublish('test-channel', data) - sleep(0.1) \ No newline at end of file + sleep(0.1) + +Async implementation +-------------------- + +`MultiDBClient` is available with async API, which looks exactly as it's sync +analogue. The core difference is that it fully relies on `EventLoop` instead of +`threading` module. + +Async client comes with async context manager support and is recommended for +graceful task cancelling. + +.. code:: python + + async with MultiDBClient(client_config) as client: + await client.set('key', 'value') + return await client.get('key') \ No newline at end of file From 4cb18ef80c625925848a34eae18100921c0d5b3f Mon Sep 17 00:00:00 2001 From: vladvildanov Date: Wed, 17 Sep 2025 12:03:38 +0300 Subject: [PATCH 6/6] Extended list of words --- .github/wordlist.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/wordlist.txt b/.github/wordlist.txt index 48ea9e8737..ab209d34be 100644 --- a/.github/wordlist.txt +++ b/.github/wordlist.txt @@ -3,6 +3,7 @@ ARGV BFCommands balancer CacheImpl +cancelling CAS CFCommands CMSCommands @@ -21,6 +22,8 @@ failover FQDN Grokzen's Healthcheck +HealthCheckPolicies +healthcheck healthchecks INCR IOError