diff --git a/docs/user-guide/deployments-administration/configuration.md b/docs/user-guide/deployments-administration/configuration.md index 953ec32e8..0c75f2073 100644 --- a/docs/user-guide/deployments-administration/configuration.md +++ b/docs/user-guide/deployments-administration/configuration.md @@ -364,7 +364,8 @@ Read [Performance Tuning Tips](/user-guide/deployments-administration/performanc ### WAL options -GreptimeDB supports two WAL storage options—Local WAL and Remote WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the two. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md) and [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) documentation. +GreptimeDB supports three WAL storage options—Local WAL, Remote WAL, and Noop WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the options. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md), [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md), and [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) documentation. + ### Logging options diff --git a/docs/user-guide/deployments-administration/wal/local-wal.md b/docs/user-guide/deployments-administration/wal/local-wal.md index 0d050dffa..0863bcfca 100644 --- a/docs/user-guide/deployments-administration/wal/local-wal.md +++ b/docs/user-guide/deployments-administration/wal/local-wal.md @@ -22,7 +22,7 @@ If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm | Configuration Option | Description | Default Value | | -------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------- | -| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage) or `kafka` (remote WAL storage in Kafka) | `"raft_engine"` | +| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage), `kafka` (remote WAL storage in Kafka), or `noop` (no-op WAL provider) | `"raft_engine"` | | `dir` | The directory where to write logs | `{data_home}/wal` | | `file_size` | The size of single WAL log file | `128MB` | | `purge_threshold` | The threshold of the WAL size to trigger purging | `1GB` | diff --git a/docs/user-guide/deployments-administration/wal/noop-wal.md b/docs/user-guide/deployments-administration/wal/noop-wal.md new file mode 100644 index 000000000..5b7520467 --- /dev/null +++ b/docs/user-guide/deployments-administration/wal/noop-wal.md @@ -0,0 +1,41 @@ +--- +keywords: [Configuration, Noop WAL, GreptimeDB Datanode, GreptimeDB, cluster mode] +description: This section describes how to configure the Noop WAL for GreptimeDB Datanode component in cluster mode. +--- +# Noop WAL + +Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data. + +## Availability + +Noop WAL is **only available in cluster mode**, not in standalone mode. + +## Use Cases + +- **Temporary WAL Unavailability**: When the WAL provider (e.g., Kafka) is temporarily unavailable, switch the Datanode to Noop WAL to keep the cluster running. +- **Testing and Development**: Useful for testing scenarios where WAL persistence is not required. + +## Data Loss Warning + +**When using Noop WAL, all unflushed data will be lost when the Datanode is shutdown or restarted.** Only use this provider temporarily when the normal WAL provider is unavailable. Not recommended for production use except in emergency situations. + +## Configuration + +To configure Noop WAL for a Datanode: + +```toml +[wal] +provider = "noop" +``` + +In a GreptimeDB cluster, WAL configuration has two parts: + +- **Metasrv** - Generates WAL metadata for new regions. Should be set to `raft_engine` or `kafka`. +- **Datanode** - Reads and writes WAL data. Configure as `noop` when the WAL provider is unavailable. + +Note: Noop WAL can only be configured on Datanode, not on Metasrv. When using Noop WAL on Datanode, Metasrv continues using its configured WAL provider. + +## Best Practices + +- Flush regions regularly using `admin flush_table()` or `admin flush_region()` to minimize data loss. +- Switch back to the normal WAL provider as soon as it becomes available. diff --git a/docs/user-guide/deployments-administration/wal/overview.md b/docs/user-guide/deployments-administration/wal/overview.md index 7c147a3f6..5f05ae439 100644 --- a/docs/user-guide/deployments-administration/wal/overview.md +++ b/docs/user-guide/deployments-administration/wal/overview.md @@ -4,11 +4,13 @@ description: This section describes the WAL (Write-Ahead Logging) in GreptimeDB, --- # Overview -The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides two WAL storage options: +The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides three WAL storage options: - **Local WAL**: Uses an embedded storage engine([raft-engine](https://github.com/tikv/raft-engine)) within the [Datanode](/user-guide/concepts/why-greptimedb.md). -- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component. +- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component. + +- **Noop WAL**: A no-op WAL provider for emergency situations when WAL becomes unavailable. Does not store any data. ## Local WAL @@ -40,9 +42,16 @@ The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) - **Network overhead**: Since WAL data needs to be transmitted over the network, careful planning of cluster network bandwidth is required to ensure low latency and high throughput, especially under write-heavy workloads. +## Noop WAL + +Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data and is only available in cluster mode. + +For detailed configuration, see the [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) page. ## Next steps - To configure the Local WAL storage, please refer to [Local WAL](/user-guide/deployments-administration/wal/local-wal.md). -- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md). \ No newline at end of file +- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md). + +- To learn more about the Noop WAL, please refer to [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md). \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md index 145b5d0ec..6a89605f0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md @@ -356,7 +356,8 @@ write_cache_size = "10GiB" ### WAL 选项 -GreptimeDB 支持两种 WAL 存储方式：本地 WAL 和 Remote WAL。关于它们的对比，请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md) 和 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) 文档。 +GreptimeDB 支持三种 WAL 存储方式：本地 WAL、Remote WAL 和 Noop WAL。关于它们的对比，请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)、[Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) 和 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) 文档。 + ### Logging 选项 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/local-wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/local-wal.md index 7ab12b00f..48cd38726 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/local-wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/local-wal.md @@ -23,7 +23,7 @@ sync_write = false | 配置项 | 描述 | 默认值 | | ----------------- | ----------------------------------------------------------------------------------------------- | ----------------- | -| `provider` | WAL 的提供者。可选项：`raft_engine`（本地文件系统存储）或 `kafka`（使用 Kafka 的远程 WAL 存储） | `"raft_engine"` | +| `provider` | WAL 的提供者。可选项：`raft_engine`（本地文件系统存储）、`kafka`（使用 Kafka 的远程 WAL 存储）或 `noop`（无操作 WAL 提供者） | `"raft_engine"` | | `dir` | 日志写入目录 | `{data_home}/wal` | | `file_size` | 单个 WAL 日志文件的大小 | `128MB` | | `purge_threshold` | 触发清理的 WAL 总大小阈值 | `1GB` | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/noop-wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/noop-wal.md new file mode 100644 index 000000000..1953f2a19 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/noop-wal.md @@ -0,0 +1,41 @@ +--- +keywords: [配置, Noop WAL, GreptimeDB Datanode, GreptimeDB, 集群模式] +description: 介绍如何在集群模式下为 GreptimeDB Datanode 组件配置 Noop WAL。 +--- +# Noop WAL + +Noop WAL 是一种特殊的 WAL 提供者，用于 WAL 暂时不可用时的紧急情况。它不会存储任何 WAL 数据。 + +## 可用性 + +Noop WAL **仅在集群模式下可用**，单机模式不支持。 + +## 使用场景 + +- **WAL 暂时不可用**：当 WAL 提供者（如 Kafka）暂时不可用时，可以将 Datanode 切换到 Noop WAL 保持集群运行。 +- **测试和开发**：适用于不需要 WAL 持久化的测试场景。 + +## 数据丢失警告 + +**使用 Noop WAL 时，Datanode 关闭或重启会导致所有未刷新的数据丢失。** 仅应在 WAL 提供者不可用时临时使用，不建议用于生产环境，除非是紧急情况。 + +## 配置 + +为 Datanode 配置 Noop WAL： + +```toml +[wal] +provider = "noop" +``` + +在 GreptimeDB 集群中，WAL 配置分为两部分： + +- **Metasrv** - 负责为新 Region 生成 WAL 元数据，应配置为 `raft_engine` 或 `kafka`。 +- **Datanode** - 负责 WAL 数据读写，可在 WAL 提供者不可用时配置为 `noop`。 + +注意：Noop WAL 只能配置在 Datanode 上，Metasrv 不支持。使用 Noop WAL 时，Metasrv 仍使用原配置的 WAL 提供者。 + +## 最佳实践 + +- 定期使用 `admin flush_table()` 或 `admin flush_region()` 刷新 Region，减少数据丢失。 +- WAL 提供者恢复后，尽快切换回正常配置。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/overview.md index af438054e..4cdd37ea8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/overview.md @@ -4,13 +4,15 @@ description: 介绍 GreptimeDB 中的 WAL（预写日志），包括本地 WAL --- # 概述 -[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一，负责持久化记录每次数据修改操作，以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持两种 WAL 存储方案： +[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一，负责持久化记录每次数据修改操作，以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持三种 WAL 存储方案： - **本地 WAL**: 使用嵌入式存储引擎 [raft-engine](https://github.com/tikv/raft-engine) ，直接集成在 [Datanode](/user-guide/concepts/why-greptimedb.md) 服务中。 - **Remote WAL**: 使用 [Apache Kafka](https://kafka.apache.org/) 作为外部的 WAL 存储组件。 +- **Noop WAL**: 无操作 WAL 提供者，用于 WAL 暂时不可用的紧急情况，不存储任何数据。 + ## 本地 WAL ### 优点 @@ -34,7 +36,7 @@ description: 介绍 GreptimeDB 中的 WAL（预写日志），包括本地 WAL - **低 RTO**: 通过将 WAL 与 Datanode 解耦，[恢复时间目标](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) (RTO) 得以最小化。当 Datanode 崩溃时，Metasrv 会发起 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) ，将受影响 Region 迁移至健康节点，无需本地重放 WAL。 -- **多消费者订阅**:Remote WAL 支持多个消费者同时订阅 WAL 日志，实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能，提升系统的高可用性和灵活性。 +- **多消费者订阅**: Remote WAL 支持多个消费者同时订阅 WAL 日志，实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能，提升系统的高可用性和灵活性。 ### 缺点 @@ -43,9 +45,16 @@ description: 介绍 GreptimeDB 中的 WAL（预写日志），包括本地 WAL - **网络开销**: WAL 数据需通过网络传输，需合理规划集群网络带宽，确保低延迟与高吞吐量，尤其在写入密集型负载下。 +## Noop WAL + +Noop WAL 是一种特殊的 WAL 提供者，用于 WAL 暂时不可用的紧急情况。它不会存储任何 WAL 数据，仅在集群模式下可用。 + +详细配置说明请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)。 ## 后续步骤 - 如需配置本地 WAL 存储，请参阅[本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)。 -- 想了解更多 Remote WAL 相关信息，请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)。 \ No newline at end of file +- 想了解更多 Remote WAL 相关信息，请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)。 + +- 想了解更多 Noop WAL 相关信息，请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)。 \ No newline at end of file diff --git a/sidebars.ts b/sidebars.ts index 9d9728447..5f057bbf3 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -313,6 +313,7 @@ const sidebars: SidebarsConfig = { 'user-guide/deployments-administration/wal/remote-wal/manage-kafka', ] }, + 'user-guide/deployments-administration/wal/noop-wal', ], }, 'user-guide/deployments-administration/configuration',