Skip to content

Commit fc1ffb6

Browse files
Updating Fusion thread information (#8554)
2 parents 5b11b1c + 6b19288 commit fc1ffb6

File tree

4 files changed

+24
-12
lines changed

4 files changed

+24
-12
lines changed

website/docs/docs/dbt-versions/core-upgrade/03-upgrading-to-fusion.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ displayed_sidebar: "docs"
77

88
import FusionAdapters from '/snippets/_fusion-dwh.md';
99
import FusionUpgradeSteps from '/snippets/_fusion-upgrade-steps.md';
10-
import FusionLifecycle from '/snippets/_fusion-lifecycle-callout.md'
10+
import FusionLifecycle from '/snippets/_fusion-lifecycle-callout.md';
11+
import FusionThreads from '/snippets/_fusion-threads.md';
1112

1213
<FusionLifecycle />
1314

@@ -238,7 +239,11 @@ In Fusion, `dbt build` runs _all_ of the unit tests _first_, and then build the
238239

239240
dbt Core runs with `--threads 1` by default. You can increase this number to run more nodes in parallel on the remote data platform, up to the max parallelism enabled by the DAG.
240241

241-
In Fusion, if `--threads` is not set, or set to `--threads 0`, dbt will use a per-adapter default value for maximum threads. Some data platforms can handle more concurrent connections than others. If there is a user-configured value for `--threads` (via CLI flag or `profiles.yml`), Fusion will use it.
242+
<Constant name="fusion"/> handles threading differently depending on your data platform:
243+
244+
<FusionThreads />
245+
246+
For more information, refer to [Using threads](/docs/running-a-dbt-project/using-threads#fusion-engine-thread-optimization).
242247

243248
#### Continue to compile unrelated nodes after hitting a compile error
244249

website/docs/docs/fusion/about-fusion.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,14 @@ Fusion also enables more-efficient deployments of large DAGs. By tracking which
4545

4646
### Thread management
4747

48-
The <Constant name="fusion_engine" /> manages parallelism differently than <Constant name="core" />. Rather than treating the `threads` setting as a strict limit on concurrent operations, Fusion dynamically optimizes parallelism based on the selected warehouse.
48+
The <Constant name="fusion_engine" /> manages parallelism differently than <Constant name="core" />. Rather than treating the `threads` setting as a strict limit on concurrent operations, Fusion optimizes parallelism based on each adapter's characteristics.
4949

50-
In Redshift, the `threads` setting limits the number of queries or statements that can run in parallel, which is important for managing Redshift's concurrency limits. In other warehouses, Fusion dynamically adjusts thread usage based on each warehouse's capabilities, using your thread configuration as guidance while automatically optimizing for maximum efficiency.
50+
- **Snowflake and Databricks**: Fusion ignores user-set threads and automatically optimizes parallelism for maximum performance.
51+
- **BigQuery and Redshift**: Fusion respects user-set threads to manage rate limits and concurrency constraints.
5152

52-
For more information, refer to [Using threads](/docs/running-a-dbt-project/using-threads#fusion-engine-thread-behavior).
53+
For BigQuery and Redshift, setting `--threads 0` or omitting the setting allows Fusion to dynamically optimize. Low thread values can significantly slow down performance on these platforms.
54+
55+
For more information, refer to [Using threads](/docs/running-a-dbt-project/using-threads#fusion-engine-thread-optimization).
5356

5457
### How to use Fusion
5558

website/docs/docs/running-a-dbt-project/using-threads.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ sidebar_label: "Use threads"
55
description: "Understand what threads mean and how to use them."
66
pagination_next: null
77
---
8+
import FusionThreads from '/snippets/_fusion-threads.md';
89

910
When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project.
1011

@@ -25,17 +26,14 @@ You will define the number of threads in your `profiles.yml` file (when developi
2526

2627
## Fusion engine thread optimization
2728

28-
The <Constant name="fusion_engine" /> handles threading differently than <Constant name="core" />. The legacy `threads` setting doesn't act as a strict limit on the number of threads created. Instead, Fusion manages parallelism dynamically based on the selected warehouse.
29+
The <Constant name="fusion_engine" /> handles threading differently than <Constant name="core" />. Rather than treating `threads` as a strict limit, <Constant name="fusion"/> manages parallelism based on each adapter's characteristics.
2930

30-
### Redshift
31+
### Adapter-specific behavior
3132

32-
In Redshift, the `threads` setting limits the number of queries or statements that can run in parallel. This behavior is the same as <Constant name="core" />, and is important for managing Redshift's concurrency limits and query queue behavior.
3333

34-
### Other warehouses
34+
<FusionThreads />
3535

36-
In other warehouses, Fusion dynamically optimizes thread usage based on the DAG and warehouse concurrency properties.
37-
38-
For more information about Fusion's approach to parallelism, refer to [the dbt Fusion engine](/docs/fusion) page.
36+
For more information about <Constant name="fusion"/>'s approach to parallelism, refer to [the <Constant name="fusion_engine"/>](/docs/fusion) page.
3937

4038
</VersionBlock>
4139

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
| Adapter | Behavior |
2+
|---------|----------|
3+
| **Snowflake** | Fusion ignores user-set threads and automatically optimizes parallelism for maximum performance. <br></br> The only supported override is `threads: 1`, which can also help resolve timeout issues if set.
4+
| **Databricks** | Fusion ignores user-set threads and automatically optimizes parallelism for maximum performance. <br></br> The only supported override is `threads: 1`, which can also help resolve timeout issues if set. |
5+
| **BigQuery** | Fusion respects user-set threads to manage API rate limits. <br></br> Setting `--threads 0` (or omitting the setting) allows <Constant name="fusion"/> to dynamically optimize parallelism. |
6+
| **Redshift** | Fusion respects user-set threads to manage concurrency limits.<br></br> Setting `--threads 0` (or omitting the setting) allows <Constant name="fusion"/> to dynamically optimize parallelism. |

0 commit comments

Comments
 (0)