Skip to content

Commit baffb4d

Browse files
CopilotWenyXu
andauthored
docs: Add Noop WAL provider documentation for nightly version (#2180)
Co-authored-by: WenyXu <[email protected]>
1 parent b58991d commit baffb4d

File tree

9 files changed

+113
-10
lines changed

9 files changed

+113
-10
lines changed

docs/user-guide/deployments-administration/configuration.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,8 @@ Read [Performance Tuning Tips](/user-guide/deployments-administration/performanc
365365

366366
### WAL options
367367

368-
GreptimeDB supports two WAL storage options—Local WAL and Remote WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the two. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md) and [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) documentation.
368+
GreptimeDB supports three WAL storage options—Local WAL, Remote WAL, and Noop WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the options. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md), [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md), and [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) documentation.
369+
369370

370371
### Logging options
371372

docs/user-guide/deployments-administration/wal/local-wal.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm
2222

2323
| Configuration Option | Description | Default Value |
2424
| -------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------- |
25-
| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage) or `kafka` (remote WAL storage in Kafka) | `"raft_engine"` |
25+
| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage), `kafka` (remote WAL storage in Kafka), or `noop` (no-op WAL provider) | `"raft_engine"` |
2626
| `dir` | The directory where to write logs | `{data_home}/wal` |
2727
| `file_size` | The size of single WAL log file | `128MB` |
2828
| `purge_threshold` | The threshold of the WAL size to trigger purging | `1GB` |
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
keywords: [Configuration, Noop WAL, GreptimeDB Datanode, GreptimeDB, cluster mode]
3+
description: This section describes how to configure the Noop WAL for GreptimeDB Datanode component in cluster mode.
4+
---
5+
# Noop WAL
6+
7+
Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data.
8+
9+
## Availability
10+
11+
Noop WAL is **only available in cluster mode**, not in standalone mode.
12+
13+
## Use Cases
14+
15+
- **Temporary WAL Unavailability**: When the WAL provider (e.g., Kafka) is temporarily unavailable, switch the Datanode to Noop WAL to keep the cluster running.
16+
- **Testing and Development**: Useful for testing scenarios where WAL persistence is not required.
17+
18+
## Data Loss Warning
19+
20+
**When using Noop WAL, all unflushed data will be lost when the Datanode is shutdown or restarted.** Only use this provider temporarily when the normal WAL provider is unavailable. Not recommended for production use except in emergency situations.
21+
22+
## Configuration
23+
24+
To configure Noop WAL for a Datanode:
25+
26+
```toml
27+
[wal]
28+
provider = "noop"
29+
```
30+
31+
In a GreptimeDB cluster, WAL configuration has two parts:
32+
33+
- **Metasrv** - Generates WAL metadata for new regions. Should be set to `raft_engine` or `kafka`.
34+
- **Datanode** - Reads and writes WAL data. Configure as `noop` when the WAL provider is unavailable.
35+
36+
Note: Noop WAL can only be configured on Datanode, not on Metasrv. When using Noop WAL on Datanode, Metasrv continues using its configured WAL provider.
37+
38+
## Best Practices
39+
40+
- Flush regions regularly using `admin flush_table()` or `admin flush_region()` to minimize data loss.
41+
- Switch back to the normal WAL provider as soon as it becomes available.

docs/user-guide/deployments-administration/wal/overview.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,13 @@ description: This section describes the WAL (Write-Ahead Logging) in GreptimeDB,
44
---
55
# Overview
66

7-
The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides two WAL storage options:
7+
The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides three WAL storage options:
88

99
- **Local WAL**: Uses an embedded storage engine([raft-engine](https://github.com/tikv/raft-engine)) within the [Datanode](/user-guide/concepts/why-greptimedb.md).
1010

11-
- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component.
11+
- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component.
12+
13+
- **Noop WAL**: A no-op WAL provider for emergency situations when WAL becomes unavailable. Does not store any data.
1214

1315
## Local WAL
1416

@@ -40,9 +42,16 @@ The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL)
4042

4143
- **Network overhead**: Since WAL data needs to be transmitted over the network, careful planning of cluster network bandwidth is required to ensure low latency and high throughput, especially under write-heavy workloads.
4244

45+
## Noop WAL
46+
47+
Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data and is only available in cluster mode.
48+
49+
For detailed configuration, see the [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) page.
4350

4451
## Next steps
4552

4653
- To configure the Local WAL storage, please refer to [Local WAL](/user-guide/deployments-administration/wal/local-wal.md).
4754

48-
- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md).
55+
- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md).
56+
57+
- To learn more about the Noop WAL, please refer to [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md).

i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -357,7 +357,8 @@ write_cache_size = "10GiB"
357357

358358
### WAL 选项
359359

360-
GreptimeDB 支持两种 WAL 存储方式:本地 WAL 和 Remote WAL。关于它们的对比,请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)[Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) 文档。
360+
GreptimeDB 支持三种 WAL 存储方式:本地 WAL、Remote WAL 和 Noop WAL。关于它们的对比,请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)[Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)[Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) 文档。
361+
361362

362363
### Logging 选项
363364

i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/local-wal.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ sync_write = false
2323

2424
| 配置项 | 描述 | 默认值 |
2525
| ----------------- | ----------------------------------------------------------------------------------------------- | ----------------- |
26-
| `provider` | WAL 的提供者。可选项:`raft_engine`(本地文件系统存储)`kafka`(使用 Kafka 的远程 WAL 存储) | `"raft_engine"` |
26+
| `provider` | WAL 的提供者。可选项:`raft_engine`(本地文件系统存储)`kafka`(使用 Kafka 的远程 WAL 存储)或 `noop`(无操作 WAL 提供者| `"raft_engine"` |
2727
| `dir` | 日志写入目录 | `{data_home}/wal` |
2828
| `file_size` | 单个 WAL 日志文件的大小 | `128MB` |
2929
| `purge_threshold` | 触发清理的 WAL 总大小阈值 | `1GB` |
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
keywords: [配置, Noop WAL, GreptimeDB Datanode, GreptimeDB, 集群模式]
3+
description: 介绍如何在集群模式下为 GreptimeDB Datanode 组件配置 Noop WAL。
4+
---
5+
# Noop WAL
6+
7+
Noop WAL 是一种特殊的 WAL 提供者,用于 WAL 暂时不可用时的紧急情况。它不会存储任何 WAL 数据。
8+
9+
## 可用性
10+
11+
Noop WAL **仅在集群模式下可用**,单机模式不支持。
12+
13+
## 使用场景
14+
15+
- **WAL 暂时不可用**:当 WAL 提供者(如 Kafka)暂时不可用时,可以将 Datanode 切换到 Noop WAL 保持集群运行。
16+
- **测试和开发**:适用于不需要 WAL 持久化的测试场景。
17+
18+
## 数据丢失警告
19+
20+
**使用 Noop WAL 时,Datanode 关闭或重启会导致所有未刷新的数据丢失。** 仅应在 WAL 提供者不可用时临时使用,不建议用于生产环境,除非是紧急情况。
21+
22+
## 配置
23+
24+
为 Datanode 配置 Noop WAL:
25+
26+
```toml
27+
[wal]
28+
provider = "noop"
29+
```
30+
31+
在 GreptimeDB 集群中,WAL 配置分为两部分:
32+
33+
- **Metasrv** - 负责为新 Region 生成 WAL 元数据,应配置为 `raft_engine``kafka`
34+
- **Datanode** - 负责 WAL 数据读写,可在 WAL 提供者不可用时配置为 `noop`
35+
36+
注意:Noop WAL 只能配置在 Datanode 上,Metasrv 不支持。使用 Noop WAL 时,Metasrv 仍使用原配置的 WAL 提供者。
37+
38+
## 最佳实践
39+
40+
- 定期使用 `admin flush_table()``admin flush_region()` 刷新 Region,减少数据丢失。
41+
- WAL 提供者恢复后,尽快切换回正常配置。

i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/wal/overview.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,15 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL
44
---
55
# 概述
66

7-
[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一,负责持久化记录每次数据修改操作,以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持两种 WAL 存储方案:
7+
[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一,负责持久化记录每次数据修改操作,以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持三种 WAL 存储方案:
88

99

1010
- **本地 WAL**: 使用嵌入式存储引擎 [raft-engine](https://github.com/tikv/raft-engine) ,直接集成在 [Datanode](/user-guide/concepts/why-greptimedb.md) 服务中。
1111

1212
- **Remote WAL**: 使用 [Apache Kafka](https://kafka.apache.org/) 作为外部的 WAL 存储组件。
1313

14+
- **Noop WAL**: 无操作 WAL 提供者,用于 WAL 暂时不可用的紧急情况,不存储任何数据。
15+
1416
## 本地 WAL
1517

1618
### 优点
@@ -34,7 +36,7 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL
3436
- **低 RTO**: 通过将 WAL 与 Datanode 解耦,[恢复时间目标](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) (RTO) 得以最小化。当 Datanode 崩溃时,Metasrv 会发起 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) ,将受影响 Region 迁移至健康节点,无需本地重放 WAL。
3537

3638

37-
- **多消费者订阅**:Remote WAL 支持多个消费者同时订阅 WAL 日志,实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能,提升系统的高可用性和灵活性。
39+
- **多消费者订阅**: Remote WAL 支持多个消费者同时订阅 WAL 日志,实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能,提升系统的高可用性和灵活性。
3840

3941

4042
### 缺点
@@ -43,9 +45,16 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL
4345

4446
- **网络开销**: WAL 数据需通过网络传输,需合理规划集群网络带宽,确保低延迟与高吞吐量,尤其在写入密集型负载下。
4547

48+
## Noop WAL
49+
50+
Noop WAL 是一种特殊的 WAL 提供者,用于 WAL 暂时不可用的紧急情况。它不会存储任何 WAL 数据,仅在集群模式下可用。
51+
52+
详细配置说明请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)
4653

4754
## 后续步骤
4855

4956
- 如需配置本地 WAL 存储,请参阅[本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)
5057

51-
- 想了解更多 Remote WAL 相关信息,请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)
58+
- 想了解更多 Remote WAL 相关信息,请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)
59+
60+
- 想了解更多 Noop WAL 相关信息,请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)

sidebars.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -311,6 +311,7 @@ const sidebars: SidebarsConfig = {
311311
'user-guide/deployments-administration/wal/remote-wal/manage-kafka',
312312
]
313313
},
314+
'user-guide/deployments-administration/wal/noop-wal',
314315
],
315316
},
316317
'user-guide/deployments-administration/configuration',

0 commit comments

Comments
 (0)