Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,8 @@ Read [Performance Tuning Tips](/user-guide/deployments-administration/performanc

### WAL options

GreptimeDB supports two WAL storage options—Local WAL and Remote WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the two. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md) and [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) documentation.
GreptimeDB supports three WAL storage options—Local WAL, Remote WAL, and Noop WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the options. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md), [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md), and [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) documentation.


### Logging options

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm

| Configuration Option | Description | Default Value |
| -------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage) or `kafka` (remote WAL storage in Kafka) | `"raft_engine"` |
| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage), `kafka` (remote WAL storage in Kafka), or `noop` (no-op WAL provider) | `"raft_engine"` |
| `dir` | The directory where to write logs | `{data_home}/wal` |
| `file_size` | The size of single WAL log file | `128MB` |
| `purge_threshold` | The threshold of the WAL size to trigger purging | `1GB` |
Expand Down
41 changes: 41 additions & 0 deletions docs/user-guide/deployments-administration/wal/noop-wal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
keywords: [Configuration, Noop WAL, GreptimeDB Datanode, GreptimeDB, cluster mode]
description: This section describes how to configure the Noop WAL for GreptimeDB Datanode component in cluster mode.
---
# Noop WAL

Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data.

## Availability

Noop WAL is **only available in cluster mode**, not in standalone mode.

## Use Cases

- **Temporary WAL Unavailability**: When the WAL provider (e.g., Kafka) is temporarily unavailable, switch the Datanode to Noop WAL to keep the cluster running.
- **Testing and Development**: Useful for testing scenarios where WAL persistence is not required.

## Data Loss Warning

**When using Noop WAL, all unflushed data will be lost when the Datanode is shutdown or restarted.** Only use this provider temporarily when the normal WAL provider is unavailable. Not recommended for production use except in emergency situations.

## Configuration

To configure Noop WAL for a Datanode:

```toml
[wal]
provider = "noop"
```

In a GreptimeDB cluster, WAL configuration has two parts:

- **Metasrv** - Generates WAL metadata for new regions. Should be set to `raft_engine` or `kafka`.
- **Datanode** - Reads and writes WAL data. Configure as `noop` when the WAL provider is unavailable.

Note: Noop WAL can only be configured on Datanode, not on Metasrv. When using Noop WAL on Datanode, Metasrv continues using its configured WAL provider.

## Best Practices

- Flush regions regularly using `admin flush_table()` or `admin flush_region()` to minimize data loss.
- Switch back to the normal WAL provider as soon as it becomes available.
15 changes: 12 additions & 3 deletions docs/user-guide/deployments-administration/wal/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ description: This section describes the WAL (Write-Ahead Logging) in GreptimeDB,
---
# Overview

The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides two WAL storage options:
The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) is a crucial component in GreptimeDB that persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides three WAL storage options:

- **Local WAL**: Uses an embedded storage engine([raft-engine](https://github.com/tikv/raft-engine)) within the [Datanode](/user-guide/concepts/why-greptimedb.md).

- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component.
- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component.

- **Noop WAL**: A no-op WAL provider for emergency situations when WAL becomes unavailable. Does not store any data.

## Local WAL

Expand Down Expand Up @@ -40,9 +42,16 @@ The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL)

- **Network overhead**: Since WAL data needs to be transmitted over the network, careful planning of cluster network bandwidth is required to ensure low latency and high throughput, especially under write-heavy workloads.

## Noop WAL

Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data and is only available in cluster mode.

For detailed configuration, see the [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) page.

## Next steps

- To configure the Local WAL storage, please refer to [Local WAL](/user-guide/deployments-administration/wal/local-wal.md).

- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md).
- To learn more about the Remote WAL, please refer to [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md).

- To learn more about the Noop WAL, please refer to [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md).
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,8 @@ write_cache_size = "10GiB"

### WAL 选项

GreptimeDB 支持两种 WAL 存储方式:本地 WAL 和 Remote WAL。关于它们的对比,请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md) 和 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) 文档。
GreptimeDB 支持三种 WAL 存储方式:本地 WAL、Remote WAL 和 Noop WAL。关于它们的对比,请参见 [WAL 概述](/user-guide/deployments-administration/wal/overview.md)。具体配置可参考 [本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)、[Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) 和 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) 文档。


### Logging 选项

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ sync_write = false

| 配置项 | 描述 | 默认值 |
| ----------------- | ----------------------------------------------------------------------------------------------- | ----------------- |
| `provider` | WAL 的提供者。可选项:`raft_engine`(本地文件系统存储)`kafka`(使用 Kafka 的远程 WAL 存储) | `"raft_engine"` |
| `provider` | WAL 的提供者。可选项:`raft_engine`(本地文件系统存储)`kafka`(使用 Kafka 的远程 WAL 存储)或 `noop`(无操作 WAL 提供者) | `"raft_engine"` |
| `dir` | 日志写入目录 | `{data_home}/wal` |
| `file_size` | 单个 WAL 日志文件的大小 | `128MB` |
| `purge_threshold` | 触发清理的 WAL 总大小阈值 | `1GB` |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
keywords: [配置, Noop WAL, GreptimeDB Datanode, GreptimeDB, 集群模式]
description: 介绍如何在集群模式下为 GreptimeDB Datanode 组件配置 Noop WAL。
---
# Noop WAL

Noop WAL 是一种特殊的 WAL 提供者,用于 WAL 暂时不可用时的紧急情况。它不会存储任何 WAL 数据。

## 可用性

Noop WAL **仅在集群模式下可用**,单机模式不支持。

## 使用场景

- **WAL 暂时不可用**:当 WAL 提供者(如 Kafka)暂时不可用时,可以将 Datanode 切换到 Noop WAL 保持集群运行。
- **测试和开发**:适用于不需要 WAL 持久化的测试场景。

## 数据丢失警告

**使用 Noop WAL 时,Datanode 关闭或重启会导致所有未刷新的数据丢失。** 仅应在 WAL 提供者不可用时临时使用,不建议用于生产环境,除非是紧急情况。

## 配置

为 Datanode 配置 Noop WAL:

```toml
[wal]
provider = "noop"
```

在 GreptimeDB 集群中,WAL 配置分为两部分:

- **Metasrv** - 负责为新 Region 生成 WAL 元数据,应配置为 `raft_engine` 或 `kafka`。
- **Datanode** - 负责 WAL 数据读写,可在 WAL 提供者不可用时配置为 `noop`。

注意:Noop WAL 只能配置在 Datanode 上,Metasrv 不支持。使用 Noop WAL 时,Metasrv 仍使用原配置的 WAL 提供者。

## 最佳实践

- 定期使用 `admin flush_table()` 或 `admin flush_region()` 刷新 Region,减少数据丢失。
- WAL 提供者恢复后,尽快切换回正常配置。
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,15 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL
---
# 概述

[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一,负责持久化记录每次数据修改操作,以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持两种 WAL 存储方案:
[预写日志](/contributor-guide/datanode/wal.md#introduction)(WAL) 是 GreptimeDB 的关键组件之一,负责持久化记录每次数据修改操作,以确保内存中的数据在故障发生时不会丢失。GreptimeDB 支持三种 WAL 存储方案:


- **本地 WAL**: 使用嵌入式存储引擎 [raft-engine](https://github.com/tikv/raft-engine) ,直接集成在 [Datanode](/user-guide/concepts/why-greptimedb.md) 服务中。

- **Remote WAL**: 使用 [Apache Kafka](https://kafka.apache.org/) 作为外部的 WAL 存储组件。

- **Noop WAL**: 无操作 WAL 提供者,用于 WAL 暂时不可用的紧急情况,不存储任何数据。

## 本地 WAL

### 优点
Expand All @@ -34,7 +36,7 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL
- **低 RTO**: 通过将 WAL 与 Datanode 解耦,[恢复时间目标](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) (RTO) 得以最小化。当 Datanode 崩溃时,Metasrv 会发起 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) ,将受影响 Region 迁移至健康节点,无需本地重放 WAL。


- **多消费者订阅**:Remote WAL 支持多个消费者同时订阅 WAL 日志,实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能,提升系统的高可用性和灵活性。
- **多消费者订阅**: Remote WAL 支持多个消费者同时订阅 WAL 日志,实现 Region 热备和 [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md) 等功能,提升系统的高可用性和灵活性。


### 缺点
Expand All @@ -43,9 +45,16 @@ description: 介绍 GreptimeDB 中的 WAL(预写日志),包括本地 WAL

- **网络开销**: WAL 数据需通过网络传输,需合理规划集群网络带宽,确保低延迟与高吞吐量,尤其在写入密集型负载下。

## Noop WAL

Noop WAL 是一种特殊的 WAL 提供者,用于 WAL 暂时不可用的紧急情况。它不会存储任何 WAL 数据,仅在集群模式下可用。

详细配置说明请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)。

## 后续步骤

- 如需配置本地 WAL 存储,请参阅[本地 WAL](/user-guide/deployments-administration/wal/local-wal.md)。

- 想了解更多 Remote WAL 相关信息,请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)。
- 想了解更多 Remote WAL 相关信息,请参阅 [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md)。

- 想了解更多 Noop WAL 相关信息,请参阅 [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md)。
1 change: 1 addition & 0 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,7 @@ const sidebars: SidebarsConfig = {
'user-guide/deployments-administration/wal/remote-wal/manage-kafka',
]
},
'user-guide/deployments-administration/wal/noop-wal',
],
},
'user-guide/deployments-administration/configuration',
Expand Down