Merge pull request #3606 from ClickHouse/best-practices

New Best practices

Author: gingerwizard

docs/about-us/adopters.md

@@ -6,7 +6,7 @@ sidebar_position: 60
 description: 'A list of companies using ClickHouse and their success stories'
 ---
 
-The following list of companies using ClickHouse and their success stories is assembled from public sources, thus might differ from current reality. We’d appreciate it if you share the story of adopting ClickHouse in your company and [add it to the list](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/about-us/adopters.md), but please make sure you won’t have any NDA issues by doing so. Providing updates with publications from other companies is also useful.
+The following list of companies using ClickHouse and their success stories is assembled from public sources, thus might differ from current reality. We'd appreciate it if you share the story of adopting ClickHouse in your company and [add it to the list](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/about-us/adopters.md), but please make sure you won't have any NDA issues by doing so. Providing updates with publications from other companies is also useful.
 
 <div class="adopters-table">
 

docs/about-us/distinctive-features.md

@@ -78,7 +78,7 @@ ClickHouse provides various ways to trade accuracy for performance:
 
 ## Adaptive Join Algorithm {#adaptive-join-algorithm}
 
-ClickHouse adaptively chooses how to [JOIN](../sql-reference/statements/select/join.md) multiple tables, by preferring hash-join algorithm and falling back to the merge-join algorithm if there’s more than one large table.
+ClickHouse adaptively chooses how to [JOIN](../sql-reference/statements/select/join.md) multiple tables, by preferring hash-join algorithm and falling back to the merge-join algorithm if there's more than one large table.
 
 ## Data Replication and Data Integrity Support {#data-replication-and-data-integrity-support}
 

docs/about-us/history.md

@@ -37,7 +37,7 @@ There is a widespread opinion that to calculate statistics effectively, you must
 However data aggregation comes with a lot of limitations:
 
 - You must have a pre-defined list of required reports.
-- The user can’t make custom reports.
+- The user can't make custom reports.
 - When aggregating over a large number of distinct keys, the data volume is barely reduced, so aggregation is useless.
 - For a large number of reports, there are too many aggregation variations (combinatorial explosion).
 - When aggregating keys with high cardinality (such as URLs), the volume of data is not reduced by much (less than twofold).

docs/about-us/support.md

@@ -28,7 +28,7 @@ Customers can only log Severity 3 tickets for single replica services across tie
 You can also subscribe to our [status page](https://status.clickhouse.com) to get notified quickly about any incidents affecting our platform.
 
 :::note
-Please note that only Subscription Customers have a Service Level Agreement on Support Incidents. If you are not currently a ClickHouse Cloud user – while we will try to answer your question, we’d encourage you to go instead to one of our Community resources:
+Please note that only Subscription Customers have a Service Level Agreement on Support Incidents. If you are not currently a ClickHouse Cloud user – while we will try to answer your question, we'd encourage you to go instead to one of our Community resources:
 
 - [ClickHouse Community Slack Channel](https://clickhouse.com/slack)
 - [Other Community Options](https://github.com/ClickHouse/ClickHouse/blob/master/README.md#useful-links)

docs/architecture/cluster-deployment.md

@@ -3,12 +3,12 @@ slug: /architecture/cluster-deployment
 sidebar_label: 'Cluster Deployment'
 sidebar_position: 100
 title: 'Cluster Deployment'
-description: 'By going through this tutorial, you’ll learn how to set up a simple ClickHouse cluster.'
+description: 'By going through this tutorial, you'll learn how to set up a simple ClickHouse cluster.'
 ---
 
 This tutorial assumes you've already set up a [local ClickHouse server](../getting-started/install.md)
 
-By going through this tutorial, you’ll learn how to set up a simple ClickHouse cluster. It’ll be small, but fault-tolerant and scalable. Then we will use one of the example datasets to fill it with data and execute some demo queries.
+By going through this tutorial, you'll learn how to set up a simple ClickHouse cluster. It'll be small, but fault-tolerant and scalable. Then we will use one of the example datasets to fill it with data and execute some demo queries.
 
 ## Cluster Deployment {#cluster-deployment}
 
@@ -19,7 +19,7 @@ This ClickHouse cluster will be a homogeneous cluster. Here are the steps:
 3.  Create local tables on each instance
 4.  Create a [Distributed table](../engines/table-engines/special/distributed.md)
 
-A [distributed table](../engines/table-engines/special/distributed.md) is a kind of "view" to the local tables in a ClickHouse cluster. A SELECT query from a distributed table executes using resources of all cluster’s shards. You may specify configs for multiple clusters and create multiple distributed tables to provide views for different clusters.
+A [distributed table](../engines/table-engines/special/distributed.md) is a kind of "view" to the local tables in a ClickHouse cluster. A SELECT query from a distributed table executes using resources of all cluster's shards. You may specify configs for multiple clusters and create multiple distributed tables to provide views for different clusters.
 
 Here is an example config for a cluster with three shards, with one replica each:
 
@@ -48,7 +48,7 @@ Here is an example config for a cluster with three shards, with one replica each
 </remote_servers>
 ```
 
-For further demonstration, let’s create a new local table with the same `CREATE TABLE` query that we used for `hits_v1` in the single node deployment tutorial, but with a different table name:
+For further demonstration, let's create a new local table with the same `CREATE TABLE` query that we used for `hits_v1` in the single node deployment tutorial, but with a different table name:
 
 ```sql
 CREATE TABLE tutorial.hits_local (...) ENGINE = MergeTree() ...
@@ -63,7 +63,7 @@ ENGINE = Distributed(perftest_3shards_1replicas, tutorial, hits_local, rand());
 
 A common practice is to create similar distributed tables on all machines of the cluster. This allows running distributed queries on any machine of the cluster. There's also an alternative option to create a temporary distributed table for a given SELECT query using [remote](../sql-reference/table-functions/remote.md) table function.
 
-Let’s run [INSERT SELECT](../sql-reference/statements/insert-into.md) into the distributed table to spread the table to multiple servers.
+Let's run [INSERT SELECT](../sql-reference/statements/insert-into.md) into the distributed table to spread the table to multiple servers.
 
 ```sql
 INSERT INTO tutorial.hits_all SELECT * FROM tutorial.hits_v1;
@@ -99,10 +99,10 @@ Here is an example config for a cluster of one shard containing three replicas:
 </remote_servers>
 ```
 
-To enable native replication [ZooKeeper](http://zookeeper.apache.org/), is required. ClickHouse takes care of data consistency on all replicas and runs a restore procedure after a failure automatically. It’s recommended to deploy the ZooKeeper cluster on separate servers (where no other processes including ClickHouse are running).
+To enable native replication [ZooKeeper](http://zookeeper.apache.org/), is required. ClickHouse takes care of data consistency on all replicas and runs a restore procedure after a failure automatically. It's recommended to deploy the ZooKeeper cluster on separate servers (where no other processes including ClickHouse are running).
 
 :::note Note
-ZooKeeper is not a strict requirement: in some simple cases, you can duplicate the data by writing it into all the replicas from your application code. This approach is **not** recommended, as in this case, ClickHouse won’t be able to guarantee data consistency on all replicas. Thus, it becomes the responsibility of your application.
+ZooKeeper is not a strict requirement: in some simple cases, you can duplicate the data by writing it into all the replicas from your application code. This approach is **not** recommended, as in this case, ClickHouse won't be able to guarantee data consistency on all replicas. Thus, it becomes the responsibility of your application.
 :::
 
 ZooKeeper locations are specified in the configuration file:

docs/best-practices/_snippets/_async_inserts.md

@@ -0,0 +1,64 @@
+import Image from '@theme/IdealImage';
+import async_inserts from '@site/static/images/bestpractices/async_inserts.png';
+
+Asynchronous inserts in ClickHouse provide a powerful alternative when client-side batching isn't feasible. This is especially valuable in observability workloads, where hundreds or thousands of agents send data continuously - logs, metrics, traces - often in small, real-time payloads. Buffering data client-side in these environments increases complexity, requiring a centralized queue to ensure sufficiently large batches can be sent.
+
+:::note
+Sending many small batches in synchronous mode is not recommended, leading to many parts being created. This will lead to poor query performance and ["too many part"](/knowledgebase/exception-too-many-parts) errors.
+:::
+
+Asynchronous inserts shift batching responsibility from the client to the server by writing incoming data to an in-memory buffer, then flushing it to storage based on configurable thresholds. This approach significantly reduces part creation overhead, lowers CPU usage, and ensures ingestion remains efficient - even under high concurrency.
+
+The core behavior is controlled via the [`async_insert`](/operations/settings/settings#async_insert) setting.
+
+<Image img={async_inserts} size="lg" alt="Async inserts"/>
+
+When enabled (1), inserts are buffered and only written to disk once one of the flush conditions is met: 
+
+(1) the buffer reaches a specified size (async_insert_max_data_size)
+(2) a time threshold elapses (async_insert_busy_timeout_ms) or 
+(3) a maximum number of insert queries accumulate (async_insert_max_query_number). 
+
+This batching process is invisible to clients and helps ClickHouse efficiently merge insert traffic from multiple sources. However, until a flush occurs, the data cannot be queried. Importantly, there are multiple buffers per insert shape and settings combination, and in clusters, buffers are maintained per node - enabling fine-grained control across multi-tenant environments. Insert mechanics are otherwise identical to those described for [synchronous inserts](/best-practices/selecting-an-insert-strategy#synchronous-inserts-by-default).
+
+### Choosing a Return Mode {#choosing-a-return-mode}
+
+The behavior of asynchronous inserts is further refined using the [`wait_for_async_insert`](/operations/settings/settings#wait_for_async_insert) setting. 
+
+When set to 1 (the default), ClickHouse only acknowledges the insert after the data is successfully flushed to disk. This ensures strong durability guarantees and makes error handling straightforward: if something goes wrong during the flush, the error is returned to the client. This mode is recommended for most production scenarios, especially when insert failures must be tracked reliably. 
+
+[Benchmarks](https://clickhouse.com/blog/asynchronous-data-inserts-in-clickhouse) show it scales well with concurrency - whether you're running 200 or 500 clients- thanks to adaptive inserts and stable part creation behavior.
+
+Setting `wait_for_async_insert = 0` enables "fire-and-forget" mode. Here, the server acknowledges the insert as soon as the data is buffered, without waiting for it to reach storage. 
+
+This offers ultra-low-latency inserts and maximal throughput, ideal for high-velocity, low-criticality data. However, this comes with trade-offs: there's no guarantee the data will be persisted, errors may only surface during flush, and it's difficult to trace failed inserts. Use this mode only if your workload can tolerate data loss. 
+
+[Benchmarks also demonstrate](https://clickhouse.com/blog/asynchronous-data-inserts-in-clickhouse) substantial part reduction and lower CPU usage when buffer flushes are infrequent (e.g. every 30 seconds), but the risk of silent failure remains.
+
+Our strong recommendation is to use `async_insert=1,wait_for_async_insert=1` if using asynchronous inserts. Using `wait_for_async_insert=0` is very risky because your INSERT client may not be aware if there are errors, and also can cause potential overload if your client continues to write quickly in a situation where the ClickHouse server needs to slow down the writes and create some backpressure in order to ensure reliability of the service.
+
+### Deduplication and reliability {#deduplication-and-reliability}
+
+By default, ClickHouse performs automatic deduplication for synchronous inserts, which makes retries safe in failure scenarios. However, this is disabled for asynchronous inserts unless explicitly enabled (this should not be enabled if you have dependent materialized views - [see issue](https://github.com/ClickHouse/ClickHouse/issues/66003)). 
+
+In practice, if deduplication is turned on and the same insert is retried - due to, for instance, a timeout or network drop - ClickHouse can safely ignore the duplicate. This helps maintain idempotency and avoids double-writing data. Still, it's worth noting that insert validation and schema parsing happen only during buffer flush - so errors (like type mismatches) will only surface at that point.
+
+### Enabling asynchronous inserts {#enabling-asynchronous-inserts}
+
+Asynchronous inserts can be enabled for a particular user, or for a specific query:
+
+- Enabling asynchronous inserts at the user level.  This example uses the user `default`, if you create a different user then substitute that username:
+  ```sql
+  ALTER USER default SETTINGS async_insert = 1
+  ```
+- You can specify the asynchronous insert settings by using the SETTINGS clause of insert queries:
+  ```sql
+  INSERT INTO YourTable SETTINGS async_insert=1, wait_for_async_insert=1 VALUES (...)
+  ```
+- You can also specify asynchronous insert settings as connection parameters when using a ClickHouse programming language client.
+
+  As an example, this is how you can do that within a JDBC connection string when you use the ClickHouse Java JDBC driver for connecting to ClickHouse Cloud :
+  ```bash
+  "jdbc:ch://HOST.clickhouse.cloud:8443/?user=default&password=PASSWORD&ssl=true&custom_http_params=async_insert=1,wait_for_async_insert=1"
+  ```
+

docs/best-practices/_snippets/_avoid_mutations.md

@@ -0,0 +1,13 @@
+In ClickHouse, **mutations** refer to operations that modify or delete existing data in a table - typically using `ALTER TABLE ... DELETE` or `ALTER TABLE ... UPDATE`. While these statements may appear similar to standard SQL operations, they are fundamentally different under the hood. 
+
+Rather than modifying rows in place, mutations in ClickHouse are asynchronous background processes that rewrite entire [data parts](/parts) affected by the change. This approach is necessary due to ClickHouse's column-oriented, immutable storage model, but it can lead to significant I/O and resource usage.
+
+When a mutation is issued, ClickHouse schedules the creation of new **mutated parts**, leaving the original parts untouched until the new ones are ready. Once ready, the mutated parts atomically replace the originals. However, because the operation rewrites entire parts, even a minor change (such as updating a single row) may result in large-scale rewrites and excessive write amplification. 
+
+For large datasets, this can produce a substantial spike in disk I/O and degrade overall cluster performance. Unlike merges, mutations can't be rolled back once submitted and will continue to execute even after server restarts unless explicitly cancelled - see [`KILL MUTATION`](/sql-reference/statements/kill#kill-mutation).
+
+Mutations are **totally ordered**: they apply to data inserted before the mutation was issued, while newer data remains unaffected. They do not block inserts but can still overlap with other ongoing queries. A SELECT running during a mutation may read a mix of mutated and unmutated parts, which can lead to inconsistent views of the data during execution. ClickHouse executes mutations in parallel per part, which can further intensify memory and CPU usage, especially when complex subqueries (like x IN (SELECT ...)) are involved.
+
+As a rule, **avoid frequent or large-scale mutations**, especially on high-volume tables. Instead, use alternative table engines such as [ReplacingMergeTree](/guides/replacing-merge-tree) or [CollapsingMergeTree](/engines/table-engines/mergetree-family/collapsingmergetree), which are designed to handle data corrections more efficiently at query time or during merges. If mutations are absolutely necessary, monitor them carefully using the system.mutations table and use `KILL MUTATION` if a process is stuck or misbehaving. Misusing mutations can lead to degraded performance, excessive storage churn, and potential service instability—so apply them with caution and sparingly.
+
+For deleting data, users can also consider [Lightweight deletes](/guides/developer/lightweight-delete) or the management of data through [partitions](/best-practices/choosing-a-partitioning-key) which allow entire parts to be [dropped efficiently](/sql-reference/statements/alter/partition#drop-partitionpart).

docs/cloud/bestpractices/avoidnullablecolumns.md renamed to docs/best-practices/_snippets/_avoid_nullable_columns.md

@@ -1,11 +1,4 @@
----
-slug: /cloud/bestpractices/avoid-nullable-columns
-sidebar_label: 'Avoid Nullable Columns'
-title: 'Avoid Nullable Columns'
-description: 'Page describing why you should avoid Nullable columns'
----
-
-[`Nullable` column](/sql-reference/data-types/nullable/) (e.g. `Nullable(String)`) creates a separate column of `UInt8` type. This additional column has to be processed every time a user works with a nullable column. This leads to additional storage space used and almost always negatively affects performance.
+[`Nullable` column](/sql-reference/data-types/nullable/) (e.g. `Nullable(String)`) creates a separate column of `UInt8` type. This additional column has to be processed every time a user works with a Nullable column. This leads to additional storage space used and almost always negatively affects performance.
 
 To avoid `Nullable` columns, consider setting a default value for that column.  For example, instead of:
 
@@ -32,6 +25,4 @@ ENGINE = MergeTree
 ORDER BY x
 ```
 
-:::note
 Consider your use case, a default value may be inappropriate.
-:::