diff --git a/website/docs/docs/fusion/new-concepts.md b/website/docs/docs/fusion/new-concepts.md index 6af783ec996..107e3b265f3 100644 --- a/website/docs/docs/fusion/new-concepts.md +++ b/website/docs/docs/fusion/new-concepts.md @@ -9,221 +9,176 @@ pagination_prev: null # New concepts - - -Learn about the net-new concepts you will encounter when using Fusion. - - - - import FusionBeta from '/snippets/_fusion-beta-callout.md'; -The new dbt Fusion engine compiles and statically analyzes SQL, to provide dialect-aware validation and extract column-level lineage. This means running `dbt compile` will generate and analyze a full logical plan for all the models in a dbt project, before running any models. -To enable this capability, Fusion introduces **two new concepts**: -- **Compilation strategies** (ahead-of-time or just-in-time) to determine whether a mode is compiled before or during DAG execution. -- **Static analysis** of code (SQL) in dbt models. When turned on, Fusion will produce a logical plan, validate it, and statically analyze the model's plan to extract column-level lineage and other rich metadata. + -In dbt core, `compile` means rendering Jinja. In dbt Fusion, compile introduces a new step and analyzes the SQL. Therefore a `compile` in Fusion can be broken down into two steps: `render` (Jinja) and `analyze` (SQL). +The dbt Fusion engine fully comprehends your project's SQL, enabling advanced capabilities like dialect-aware validation and precise column-level lineage. -## Compilation strategies +It can do this because its compilation step is more comprehensive than that of the engine. When referred to _compilation_, it only meant _rendering_ — converting Jinja-templated strings into a SQL query to send to a database. -[Ahead-of-time compilation](https://en.wikipedia.org/wiki/Ahead-of-time_compilation) is the default compilation strategy of Fusion. This means when a user runs `dbt run` on Fusion, it compiles all models first, then runs them, providing a platform for significant performance improvements and guaranteeing correctness of the DAG before execution. This models compiled languages like Rust, Typescript, or Java where the compiler catches all issues upfront before anything is executed. +The dbt Fusion engine can also render Jinja, but then it completes a second phase: producing and validating with _static analysis_ a logical plan for every rendered query in the project. This static analysis step is the cornerstone of Fusion's new capabilities. - + -[Just-in-time compilation](https://en.wikipedia.org/wiki/Just-in-time_compilation) is the default behavior of . dbt compiles, then runs, each model sequentially in the DAG. This is necessary in cases where the results of upstream models are inputs to introspective queries used to template downstream models. The downstream model must be compiled "just-in-time," after the upstream models have finished building. +| Step | dbt Core engine | dbt Fusion engine | +|------|-----------------|--------------------| +| Render Jinja into SQL | ✅ | ✅ | +| Produce and statically analyze logical plan | ❌ | ✅ | +| Run rendered SQL | ✅ | ✅ | - +## Rendering strategies -### Dynamic Templating + -While Fusion defaults to ahead-of-time (AOT) compilation for performance and validation benefits, models with dynamic Jinja templating ([introspective queries](/faqs/Warehouse/db-connection-dbt-compile)) require just-in-time compilation. + + -Dynamic templating functions that depend on the state of the data platform, such as [run_query](/reference/dbt-jinja-functions/run_query) and [`dbt_utils.get_column_values`](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#get_column_values-source), need to query the database to determine what SQL to generate. + will _always_ use **Just In Time (JIT) rendering**. It renders a model, runs it in the warehouse, then moves on to the next model. -These patterns create a chicken-and-egg problem for AOT compilation: Fusion can't know what SQL will be generated until it runs the upstream models and queries their results. The downstream model's SQL depends on the data produced _at runtime_ in the upstream model, and so it must be compiled "just in time." + + -**Fusion intelligently switches to JIT compilation only for models that require it, while maintaining AOT compilation for all other models in your DAG.** This selective approach gives you the best of both worlds: +The will _default to_ **Ahead of Time (AOT) rendering and analysis**. It renders all models in the project, then produces and statically analyzes every model's logical plan, and only then will it start running models in the warehouse. -- Models without dynamic templating benefit from AOT's performance improvements and upfront validation -- Models with dynamic templating still work correctly with JIT compilation -- The compilation mode is determined on a per-model basis, not for the entire DAG +By rendering and analyzing all models ahead of time, and only beginning execution once everything is proven to be valid, the avoids consuming any warehouse resources unnecessarily. By contrast, SQL errors in models run by 's engine will only be flagged by the database itself during execution. -For example, in a DAG like this: - +### Rendering introspective queries +The exception to AOT rendering is an introspective model: a model whose rendered SQL depends on the results of a database query. Models containg macros like `run_query()` or `dbt_utils.get_column_values()` are introspective. Introspection causes issues with ahead-of-time rendering because: -Fusion will: -1. Render `model_a`, `model_b`, `model_d` ahead of time -2. In DAG order, analyze the SQL of `model_a` then `model_b` -3. Run `model_a → model_b` -4. Switch to JIT compile-run for `model_c` (due to dynamic templating) -5. Continue with SQL analysis then run for `model_d` (since it depends on a dynamic model) +- Most introspective queries are run against the results of an earlier model in the DAG, which may not yet exist in the database during AOT rendering. +- Even if the model does exist in the database, it might be out of date until after the model has been refreshed. +The switches to **JIT rendering for introspective models**, to ensure it renders them the same way as . -However, if you have a parallel branch: - +Note that macros like `adapter.get_columns_in_relation()` and `dbt_utils.star()` _can_ be rendered and analyzed ahead of time, as long as the [`Relations`](/reference/dbt-classes#relation) they inspect aren't themselves dynamic. This is because the populates schemas into memory as part of the compilation process. +## Principles of static analysis -Fusion will: -1. AOT compile `model_a`, `model_b` _and_ `model_x`, `model_y`, `model_z` because none depend on a dynamically templated model. It also AOT renders `model_d` since its Jinja is not dynamic. -2. Switch to JIT compile and run for `model_c` -3. JIT analyze and run `model_d` +[Static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) is meant to guarantee that if a model compiles without error in development, it will also run without compilation errors when deployed. Introspective queries can break this promise by making it possible to modify the rendered query after a model is committed to source control. -This granular approach ensures maximum performance and reliability, while maintaining compatibility with dynamic SQL patterns. +The is unique in that it can statically analyze not just a single model in isolation, but every query from one end of your DAG to the other. Even your database can only validate the query in front of it! Concepts like [information flow theory](https://roundup.getdbt.com/i/156064124/beyond-cll-information-flow-theory-and-metadata-propagation) — although not incorporated into the dbt platform [yet](https://www.getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine) — rely on stable inputs and the ability to trace columns DAG-wide. -## The `static_analysis` config +### Static analysis and introspective queries -Fusion can statically analyze the vast majority of SQL in supported dialects, with some exceptions and limitations. You can toggle on and off the new static (SQL) analysis features for specific models in your project, if they contain unsupported or dynamic SQL. This configuration propagates downstream, which means if an upstream model's static analysis is turned off, then Fusion will also turn off static analysis for any downstream models referencing that model. +When Fusion encounters an introspective query, that model will switch to just-in-time rendering (as described above). Both the introspective model and all of its descendants will also be opted in to JIT static analysis. We refer to JIT static analysis as "unsafe" because it will still capture most SQL errors and prevent execution of an invalid model, but only after upstream models have already been materialized. -## Usage +This classification is meant to indicate that Fusion can no longer 100% guarantee alignment between what it analyzes and what will be executed. The most common real-world example where unsafe static analysis can cause an issue is a standalone `dbt compile` step (as opposed to the compilation that happens as part of a `dbt run`). -You can set `static_analysis` as a model-level config (preferred), or as a CLI flag for an entire run, which overrides model-level configuration and is really intended as an aid to debugging. Refer to [CLI options](/reference/global-configs/command-line-options) and [Configurations and properties](/reference/configs-and-properties) to learn more about configs. +During a `dbt run`, JIT rendering ensures the downstream model's code will be up to date with the current warehouse state, but a standalone compile does not refresh the upstream model. In this scenario Fusion will read from the upstream model as it was last run. This is _probably_ fine, but could lead to errors being raised incorrectly (a false positive) or not at all (a false negative). -The `static_analysis` model-level config uses these options: + + -- `on`: (Default) Statically analyzes SQL. -- `off`: Skips SQL analysis on a model and all downstream models that depend on it. If Fusion detects that a model is dynamically templated with introspective queries, it will automatically shut off static analysis for this model and all downstream models, and switch to just-in-time compilation. -- `unsafe`: Force Fusion to analyze the SQL for dynamically generated models. As the user, you accept the risk that SQL may _change_ between compile and run, as the state of the data platform or upstream models change. At runtime, the resulting SQL may be invalid or different. Fusion switches to JIT compilation for this model and models downstream. +You will still derive significant benefits from "unsafe" static analysis compared to no static analysis, and we recommend leaving it on unless you notice it causing you problems. Better still, you should consider whether your introspective code could be rewritten in a way that is eligible for AOT rendering and static analysis. - +## Recapping the differences between engines -```yml -version: 2 +dbt Core: -models: - - name: - config: - static_analysis: unsafe - ... -``` +- renders all models just-in-time +- never runs static analysis - +The dbt Fusion engine: -The CLI flag uses `--static-analysis=unsafe` or `--static-analysis=off` for the entire run, which takes precedence over the model-level config: +- renders all models ahead-of-time, unless they use introspective queries +- statically analyzes all models, defaulting to ahead-of-time unless they or their parents were rendered just-in-time, in which case the static analysis step will also happen just-in-time. -```bash -dbt run --static-analysis=unsafe -``` +## Configuring `static_analysis` -## Example: new concepts in action +Beyond the default behavior described above, you can always modify the way static analysis is applied for specific models in your project. Remember that **a model is only eligible for static analysis if all of its parents are also eligible.** -Imagine a DAG with `model_a → model_b → model_c → model_d`. All of these models are defined with static SQL. +The `static_analysis` options are: -### Default behavior (`static_analysis: on`) +- `on`: Statically analyze SQL. The default for non-introspective models, depends on AOT rendering. +- `unsafe`: Statically analyze SQL. The default for introspective models. Always uses JIT rendering. +- `off`: Skip SQL analysis on this model and its descendants. -- During `dbt compile`, Fusion will compile and analyze all models -- During `dbt run`, Fusion will compile and analyze all models, then run all models +When you disable static analysis, features of the VS Code extension which depend on SQL comprehension will be unavailable. -### After adding dynamic SQL +The best place to configure `static_analysis` is as a config on an individual model or group of models. As a debugging aid, you can also use the `--static-analysis off` or `--static-analysis unsafe` CLI flags to override all model-level configuration. Refer to [CLI options](/reference/global-configs/command-line-options) and [Configurations and properties](/reference/configs-and-properties) to learn more about configs. -We update `model_c` to introduce dynamic SQL (such as `dbt_utils.get_column_values`). Fusion will automatically detect that `model_c` has dynamic Jinja. It will turn off static analysis for `model_c+` and switch them to a just-in-time compilation strategy. +### Example configurations -During `dbt compile`, Fusion will: -- Parse the project, detect that `model_c` is dynamic, switch `static_analysis: off` for `model_c+` -- Compile and analyze `model_a`, `model_b` -- Skip analysis for `model_c+` (including `model_d`) +Disable static analysis for all models in a package: -During `dbt run`, Fusion will: -- Parse the project, detect that `model_c` is dynamic, switch `static_analysis: off` and just-in-time rendering for `model_c+` -- Compile and analyze `model_a`, `model_b` -- Run `model_a → model_b` -- Render SQL for `model_c` (without static analysis), then run `model_c` -- Render SQL for `model_d` (no static analysis), then run `model_d` + -### Explicitly set `static_analysis: unsafe` for `model_c` +```yml +name: jaffle_shop -This configuration tells Fusion to attempt static analysis, even though `model_c` is dynamically templated. +models: + jaffle_shop: + marts: + +materialized: table + + a_package_with_introspective_queries: + +static_analysis: off +``` -During `dbt compile`, Fusion will: -- Parse the project, detect that `model_c` is unsafe, but the explicit user configuration of `static_analysis: unsafe` tells Fusion not to shut off static analysis for `model_c+` -- Compile and analyze all models. For `model_b`'s introspection queries, it uses data from previously-built tables or production (if using defer) + -During `dbt run`, Fusion will: -- Parse the project, detect that `model_c` is unsafe, but the explicit user configuration of `static_analysis: unsafe` tells Fusion not to shut off static analysis for `model_c+`. Fusion still switches to JIT compilation for those models. -- Compile and analyze `model_a`, `model_b` -- Run `model_a → model_b` -- Compile `model_c` (including static analysis), then run `model_c` -- Compile `model_d` (including static analysis), then run `model_d` +Disable static analysis for a model using a custom UDF: -- **Warning:** The actual SQL executed for `model_c` and `model_d` may differ from what was analyzed during compilation, which may lead to schema mismatches or errors. Many of those errors will be detected during static analysis, but the detection is "just-in-time," after `model_a → model_b` have already been materialized. + -## Limitations to static analysis +```sql +{{ config(static_analysis='off') }} -Fusion is unable to compile user-defined functions (UDFs), which define custom functions in Snowflake and other data warehouses. When using a UDF in a model, you need to set `static_analysis: off`. In the future, we intend to add native support for UDF definition and compilation within Fusion. +select + user_id, + my_cool_udf(ip_address) as cleaned_ip +from {{ ref('my_model') }} +``` -Fusion automatically detects dynamically templated SQL, via introspective query calls within dbt-jinja, but it is unable to detect dynamic SQL, such as the Snowflake `PIVOT` function. You can either configure your model to set `static_analysis: off`, or you can refactor your model to have a statically enforceable schema. (See example below.) + -### Dynamic SQL +### When should I turn static analysis `off`? -We currently do not detect when the schema is dynamic based on SQL functionality (instead of Jinja). For example, when you use Snowflake with the `ANY` keyword: +Static analysis may incorrectly fail on valid queries if they contain: -```sql -with quarterly_sales as ( - select * from values - (1, 10000, '2023_Q1'), - (1, 400, '2023_Q1'), - (2, 4500, '2023_Q1'), - (2, 35000, '2023_Q1'), - (3, 10200, '2023_Q4') - as quarterly_sales(emp_id, amount, quarter) -) - -select * -from quarterly_sales -pivot ( - sum(amount) for quarter in (ANY) -) -order by emp_id -``` +- **syntax or native functions** that the doesn't recognize. Please [open an issue](https://github.com/dbt-labs/dbt-fusion/issues) in addition to disabling static analysis. +- **user-defined functions** that the doesn't recognize. You will need to temporarily disable static analysis. Native support for UDF compilation will arrive in a future version - see [dbt-fusion#69](https://github.com/dbt-labs/dbt-fusion/issues/69). +- **dynamic SQL** such as [Snowflake's PIVOT ANY](https://docs.snowflake.com/en/sql-reference/constructs/pivot#dynamic-pivot-on-all-distinct-column-values-automatically) which cannot be statically analyzed. You can disable static analysis, refactor your pivot to use explicit column names, or create a [dynamic pivot in Jinja](https://github.com/dbt-labs/dbt-utils#pivot-source). +- **highly volatile data feeding an introspective query** during a standalone `dbt compile` invocation. Because the `dbt compile` step does not run models, it uses old data or defers to a different environment when running introspective queries. The more frequently the input data changes, the more likely it is for this divergence to cause a compilation error. Consider whether these standalone `dbt compile` commands are necessary before disabling static analysis. -This example model uses a dynamic schema based on SQL functionality (not Jinja), which causes an error: +## Examples -```terminal -error: dbt0432: PIVOT ANY is not compilable - --> models/example/my_first_model.sql:12:29 (target/compiled/models/example/my_first_model.sql:12:29) -``` +### No introspective models -To fix this error, you may: -- configure your model with `static_analysis: off` -- refactor your model to use dynamic Jinja templating, e.g. `dbt_utils.get_column_values`, and configure your model with `static_analysis: unsafe` -- refactor your model to be a static pivot, and benefit from safe static analysis + + -```sql -with quarterly_sales as ( - select * from values - (1, 10000, '2023_01'), - (1, 400, '2023_01'), - (2, 4500, '2023_01'), - (2, 35000, '2023_01'), - (3, 10200, '2023_04') -as quarterly_sales(emp_id, amount, quarter) -) - -select * from quarterly_sales pivot ( -sum(amount) for quarter in ('2023_01', '2023_04')) -order by emp_id -``` +- Fusion renders each model in order. +- Then it statically analyzes each model's logical plan in order. +- Finally, it runs each model's rendered SQL. Nothing is persisted to the database until Fusion has validated the entire project. -### UDFs +### Introspective model with `unsafe` static analysis -If you call a user-defined function in your model SQL: +Imagine we update `model_c` to contain an introspective query (such as `dbt_utils.get_column_values`). We'll say it's querying `model_b`, but the 's response is the same regardless of what the introspection does. -```sql -select my_example_udf(player_id) - from {{ source('raw_data', 'raw_players') }} -``` + + -Fusion will raise an error during static analysis: +- During parsing, Fusion discovers `model_c`'s introspective query. It switches `model_c` to JIT rendering and opts `model_c+` in to JIT static analysis. +- `model_a` and `model_b` are still eligible for AOT compilation, so Fusion handles them the same as in the introspection-free example above. `model_d` is still eligible for AOT rendering (but not analysis). +- Once `model_b` is run, Fusion renders `model_c`'s SQL (using the just-refreshed data), analyzes it, and runs it. All three steps happen back-to-back. +- `model_d`'s AOT-rendered SQL is analyzed and run. -```terminal -error: dbt0209: No function MY_EXAMPLE_UDF - --> models/staging/stg_players.sql:7:9 (target/compiled/models/staging/stg_players.sql:7:9) -``` + + -To fix this error, you need to set `static_analysis: off` for that model. This will also have the effect of turning off static analysis for any downstream models as well. +As you'd expect, a branching DAG will AOT compile as much as possible before moving on to the JIT components, and will work with multiple `--threads` if they're available. Here, `model_c` can start rendering as soon as `model_b` has finished running, while the AOT-compiled `model_x` and `model_y` run separately: import AboutFusion from '/snippets/_about-fusion.md'; diff --git a/website/src/manim-visualizations/.gitignore b/website/src/manim-visualizations/.gitignore new file mode 100644 index 00000000000..8231aed488e --- /dev/null +++ b/website/src/manim-visualizations/.gitignore @@ -0,0 +1,12 @@ +# Python-generated files +__pycache__/ +*.py[oc] +build/ +dist/ +wheels/ +*.egg-info + +# Virtual environments +.venv +media/ +.DS_Store diff --git a/website/src/manim-visualizations/README.md b/website/src/manim-visualizations/README.md new file mode 100644 index 00000000000..a00abbfd18c --- /dev/null +++ b/website/src/manim-visualizations/README.md @@ -0,0 +1 @@ +[Manim](https://docs.manim.community/en/stable/) is a way to generate animations as code diff --git a/website/src/manim-visualizations/fusion_static_analysis_concepts.py b/website/src/manim-visualizations/fusion_static_analysis_concepts.py new file mode 100644 index 00000000000..5f08479626d --- /dev/null +++ b/website/src/manim-visualizations/fusion_static_analysis_concepts.py @@ -0,0 +1,483 @@ +from manim import * +import networkx as nx + +# Colors +DBT_PURPLE = ManimColor.from_hex("#8B5CF6") +BOX_COLOR = DARK_GREY +PENDING_CIRCLE_COLOR = GREY_C +AOT_CIRCLE_COLOR = DBT_PURPLE +JIT_CIRCLE_COLOR = PURE_GREEN +EXECUTE_COLOR = PURE_GREEN + +class RectangleVertex(VGroup): + def __init__(self, label=None, **rect_kwargs): + # 1) Build and stash the rectangle + rect = Rectangle(**rect_kwargs) + self.rect = rect + + # 2) Center the label on top of the rectangle + if label: + label.move_to(rect.get_center() + 0.15 * UP) + super().__init__(rect, label) + else: + super().__init__(rect) + + # 3) Add three traffic-light style indicators + circle_radius = 0.18 + spacing = 0.05 + margin = circle_radius + spacing + # Lay circles horizontally from the bottom-right corner + self.indicators = VGroup(*[ + Circle( + radius=circle_radius, + fill_color=PENDING_CIRCLE_COLOR, + fill_opacity=1.0, + stroke_width=1, + stroke_color=BOX_COLOR, + ).move_to( + rect.get_corner(DR) + + np.array([ + -margin - i * (2 * circle_radius + spacing), + margin, + 0 + ]) + ) + for i in reversed(range(3)) + ]) + self.add(self.indicators) + + def get_boundary_point(self, direction): + """ + Override the boundary point calculation to always use the rectangle's boundary, + not the VGroup's overall bounding box. This ensures arrows connect to the + rectangle edges even when the label spills out. + """ + # Use the rectangle's center and dimensions instead of the VGroup's + center = self.rect.get_center() + w = self.rect.width / 2 + h = self.rect.height / 2 + + # Calculate which edge the direction vector intersects + if abs(direction[0]) / w > abs(direction[1]) / h: + # Intersects left or right edge + sign = np.sign(direction[0]) + x = center[0] + sign * w + y = center[1] + direction[1] * (w / abs(direction[0])) if direction[0] != 0 else center[1] + else: + # Intersects top or bottom edge + sign = np.sign(direction[1]) + y = center[1] + sign * h + x = center[0] + direction[0] * (h / abs(direction[1])) if direction[1] != 0 else center[0] + + return np.array([x, y, center[2]]) + + +class BaseModelDigraph(Scene): + def __init__(self, vertices, edges, layout, color_sequence, labels=None, **kwargs): + super().__init__(**kwargs) + self.vertices = vertices + self.edges = edges + self.layout = layout + self.color_sequence = color_sequence + + # Generate labels if not provided + if labels is None: + self.labels = {v: Text(v.replace("__", "\n"), font_size=20 if "introspective" in v else 26, font="PT Mono") for v in vertices} + else: + self.labels = labels + + def center_layout(self, layout): + """Center a layout dictionary around the origin""" + positions = list(layout.values()) + + # Calculate centroid + centroid_x = sum(pos[0] for pos in positions) / len(positions) + centroid_y = sum(pos[1] for pos in positions) / len(positions) + centroid = np.array([centroid_x, centroid_y, 0]) + + # Adjust all positions by subtracting the centroid + return {vertex: pos - centroid for vertex, pos in layout.items()} + + def create_animation_for_vertex(self, graph, vertex_name, circle_index, new_color, step_index): + """Create the pulse and fill animation for a single vertex""" + vertex = graph.vertices[vertex_name] + circle = vertex.indicators[circle_index] + + # Create small text with the step number + step_text = Text( + str(step_index), + font_size=16, + color=BLACK if new_color != DBT_PURPLE else WHITE, + weight=BOLD + ).move_to(circle.get_center()) + + # Make text initially invisible and add it to the vertex group + step_text.set_opacity(0) + vertex.add(step_text) + + pulse_and_fill = Succession( + # Step 1: grow + change color together + AnimationGroup( + vertex.animate.scale(1.1), + lag_ratio=0, # run both at the same time + ), + # Step 2: shrink back, change color, and show text + AnimationGroup( + vertex.animate.scale(1), + circle.animate.set_fill(new_color), + step_text.animate.set_opacity(1), + lag_ratio=0, + ), + lag_ratio=1, # only start the shrink once the grow+fill is done + ) + + return pulse_and_fill + + def construct(self): + # Create the directed graph using our custom RectangleVertex + graph = DiGraph( + vertices=self.vertices, + edges=self.edges, + layout=self.layout, + vertex_type=RectangleVertex, + vertex_config={ + "height": 1.5, + "width": 2.5, + "fill_color": BOX_COLOR, + "fill_opacity": 1.0, + "stroke_color": BOX_COLOR, + "stroke_width": 2, + }, + labels=self.labels, + edge_config={ + "tip_length": 0.2, + "stroke_width": 2.5, + "stroke_color": BOX_COLOR, + }, + ) + + # Center the graph around the origin + graph.move_to(ORIGIN) + + self.add(graph) + + animation_duration = 0.75 # Duration per animation + total_animation_time = len(self.color_sequence) * animation_duration + target_total_time = 20.0 # Target video length in seconds + + # Find connected components in the graph + nx_graph = nx.Graph() # Use undirected graph for component analysis + nx_graph.add_nodes_from(self.vertices) + # Add edges as undirected for component detection + for edge in self.edges: + nx_graph.add_edge(edge[0], edge[1]) + + # Get connected components and create a mapping + components = list(nx.connected_components(nx_graph)) + vertex_to_component = {} + for i, component in enumerate(components): + for vertex in component: + vertex_to_component[vertex] = i + + # Animate each indicator and its rectangle in sequence or parallel + step_index = 0 + current_component = None + + for item in self.color_sequence: + # Check if this is a parallel animation (list) or sequential (tuple) + if isinstance(item, list): + # Parallel animations - all items in the list happen simultaneously + parallel_animations = [] + + for vertex_name, circle_index, new_color in item: + if new_color is not None: + # Reset step_index if we're switching to a different connected component + vertex_component = vertex_to_component[vertex_name] + if current_component != vertex_component: + current_component = vertex_component + step_index = 0 + + step_index += 1 # Only increment for actual color changes + animation = self.create_animation_for_vertex(graph, vertex_name, circle_index, new_color, step_index) + parallel_animations.append(animation) + + # Play all parallel animations together + if parallel_animations: + self.play(*parallel_animations, run_time=animation_duration) + + else: + # Sequential animation (single tuple) + vertex_name, circle_index, new_color = item + + if new_color is None: + # Just pause for the animation duration when new_color is None + self.wait(animation_duration) + else: + # Reset step_index if we're switching to a different connected component + vertex_component = vertex_to_component[vertex_name] + if current_component != vertex_component: + current_component = vertex_component + step_index = 0 + + step_index += 1 # Only increment for actual color changes + animation = self.create_animation_for_vertex(graph, vertex_name, circle_index, new_color, step_index) + self.play(animation, run_time=animation_duration) + + remaining_time = max(0, target_total_time - total_animation_time) + if remaining_time > 0: + self.wait(remaining_time) + +class FusionJitRunUnsafeComplexDag(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "model_a", + "model_b", + "model_x", + "model_y", + "introspective__model_c", + "model_d", + ] + edges = [ + ("model_a", "model_b"), + ("model_b", "model_x"), + ("model_x", "model_y"), + ("model_b", "introspective__model_c"), + ("introspective__model_c", "model_d"), + ] + layout = { + "model_a": 5.5 * LEFT + 1.5 * UP, + "model_b": 2.5 * LEFT + 1.5 * UP, + "introspective__model_c": 0.5 * RIGHT + 3 * UP, + "model_x": 0.5 * RIGHT , + "model_d": 3.5 * RIGHT + 3 * UP, + "model_y": 3.5 * RIGHT, + } + color_sequence = [ + ("model_a", 0, AOT_CIRCLE_COLOR), + ("model_b", 0, AOT_CIRCLE_COLOR), + [ + ("model_d", 0, AOT_CIRCLE_COLOR), + ("model_x", 0, AOT_CIRCLE_COLOR) + ], + ("model_y", 0, AOT_CIRCLE_COLOR), + + ("model_a", 1, AOT_CIRCLE_COLOR), + ("model_b", 1, AOT_CIRCLE_COLOR), + ("model_x", 1, AOT_CIRCLE_COLOR), + ("model_y", 1, AOT_CIRCLE_COLOR), + + ("model_a", 2, EXECUTE_COLOR), + ("model_b", 2, EXECUTE_COLOR), + + [ + ("introspective__model_c", 0, JIT_CIRCLE_COLOR), + ("model_x", 2, EXECUTE_COLOR), + ], + [ + ("introspective__model_c", 1, JIT_CIRCLE_COLOR), + ("model_y", 2, EXECUTE_COLOR), + ], + ("introspective__model_c", 2, EXECUTE_COLOR), + ("model_d", 1, JIT_CIRCLE_COLOR), + ("model_d", 2, EXECUTE_COLOR), + ] + + # Center the layout around origin + layout = self.center_layout(layout) + + super().__init__(vertices, edges, layout, color_sequence, **kwargs) + +class CoreJitRun(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "model_a", + "model_b", + "model_c", + "model_d", + ] + edges = [ + ("model_a", "model_b"), + ("model_b", "model_c"), + ("model_c", "model_d"), + ] + layout = { + "model_a": 6 * LEFT, + "model_b": 3 * LEFT, + "model_c": 0 * LEFT, + "model_d": 3 * RIGHT, + } + color_sequence = [ + # add None for the second circle so that the core and fusion animations end at the same time since they're right beside each other + ("model_a", 0, JIT_CIRCLE_COLOR), + ("model_a", 1, None), + ("model_a", 2, EXECUTE_COLOR), + ("model_b", 0, JIT_CIRCLE_COLOR), + ("model_b", 1, None), + ("model_b", 2, EXECUTE_COLOR), + ("model_c", 0, JIT_CIRCLE_COLOR), + ("model_c", 1, None), + ("model_c", 2, EXECUTE_COLOR), + ("model_d", 0, JIT_CIRCLE_COLOR), + ("model_d", 1, None), + ("model_d", 2, EXECUTE_COLOR), + ] + + super().__init__(vertices, edges, layout, color_sequence, **kwargs) + + + +class FusionAotRun(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "model_a", + "model_b", + "model_c", + "model_d", + ] + edges = [ + ("model_a", "model_b"), + ("model_b", "model_c"), + ("model_c", "model_d"), + ] + layout = { + "model_a": 6 * LEFT, + "model_b": 3 * LEFT, + "model_c": 0 * LEFT, + "model_d": 3 * RIGHT, + } + color_sequence = [ + ("model_a", 0, AOT_CIRCLE_COLOR), + ("model_b", 0, AOT_CIRCLE_COLOR), + ("model_c", 0, AOT_CIRCLE_COLOR), + ("model_d", 0, AOT_CIRCLE_COLOR), + ("model_a", 1, AOT_CIRCLE_COLOR), + ("model_b", 1, AOT_CIRCLE_COLOR), + ("model_c", 1, AOT_CIRCLE_COLOR), + ("model_d", 1, AOT_CIRCLE_COLOR), + ("model_a", 2, EXECUTE_COLOR), + ("model_b", 2, EXECUTE_COLOR), + ("model_c", 2, EXECUTE_COLOR), + ("model_d", 2, EXECUTE_COLOR), + ] + + super().__init__(vertices, edges, layout, color_sequence, **kwargs) + + + +class FusionJitRunUnsafe(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "model_a", + "model_b", + "introspective__model_c", + "model_d", + ] + edges = [ + ("model_a", "model_b"), + ("model_b", "introspective__model_c"), + ("introspective__model_c", "model_d"), + ] + layout = { + "model_a": 6 * LEFT, + "model_b": 3 * LEFT, + "introspective__model_c": 0 * LEFT, + "model_d": 3 * RIGHT, + } + color_sequence = [ + ("model_a", 0, AOT_CIRCLE_COLOR), + ("model_b", 0, AOT_CIRCLE_COLOR), + ("model_d", 0, AOT_CIRCLE_COLOR), + ("model_a", 1, AOT_CIRCLE_COLOR), + ("model_b", 1, AOT_CIRCLE_COLOR), + ("model_a", 2, EXECUTE_COLOR), + ("model_b", 2, EXECUTE_COLOR), + ("introspective__model_c", 0, JIT_CIRCLE_COLOR), + ("introspective__model_c", 1, JIT_CIRCLE_COLOR), + ("introspective__model_c", 2, EXECUTE_COLOR), + ("model_d", 1, JIT_CIRCLE_COLOR), + ("model_d", 2, EXECUTE_COLOR), + ] + + super().__init__(vertices, edges, layout, color_sequence, **kwargs) + + + +class FusionJitCompileUnsafe(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "model_a", + "model_b", + "introspective__model_c", + "model_d", + ] + edges = [ + ("model_a", "model_b"), + ("model_b", "introspective__model_c"), + ("introspective__model_c", "model_d"), + ] + layout = { + "model_a": 6 * LEFT, + "model_b": 3 * LEFT, + "introspective__model_c": 0 * LEFT, + "model_d": 3 * RIGHT, + } + color_sequence = [ + ("model_a", 0, AOT_CIRCLE_COLOR), + ("model_b", 0, AOT_CIRCLE_COLOR), + ("model_d", 0, AOT_CIRCLE_COLOR), + ("model_a", 1, AOT_CIRCLE_COLOR), + ("model_b", 1, AOT_CIRCLE_COLOR), + ("introspective__model_c", 0, JIT_CIRCLE_COLOR), + ("introspective__model_c", 1, JIT_CIRCLE_COLOR), + ("model_d", 1, JIT_CIRCLE_COLOR), + ] + + super().__init__(vertices, edges, layout, color_sequence, **kwargs) + + + + +class FusionCoreLifecycleComparisonSingleNode(BaseModelDigraph): + def __init__(self, **kwargs): + vertices = [ + "core_model_a", + "core_model_b", + "fusion_model_a", + "fusion_model_b", + ] + edges = [ + ("core_model_a", "core_model_b"), + ("fusion_model_a", "fusion_model_b"), + ] + labels = { + "core_model_a": Text("model_a\n(dbt Core)", font_size=26, font="PT Mono"), + "core_model_b": Text("model_b\n(dbt Core)", font_size=26, font="PT Mono"), + "fusion_model_a": Text("model_a\n(Fusion)", font_size=26, font="PT Mono"), + "fusion_model_b": Text("model_b\n(Fusion)", font_size=26, font="PT Mono"), + } + layout = { + "core_model_a": 4.5 * LEFT, + "core_model_b": 1.5 * LEFT, + "fusion_model_a": 1.5 * RIGHT, + "fusion_model_b": 4.5 * RIGHT, + } + color_sequence = [ + ("core_model_a", 0, JIT_CIRCLE_COLOR), + ("core_model_a", 1, None), + ("core_model_a", 2, EXECUTE_COLOR), + ("core_model_b", 0, JIT_CIRCLE_COLOR), + ("core_model_b", 1, None), + ("core_model_b", 2, EXECUTE_COLOR), + + ("fusion_model_a", 0, AOT_CIRCLE_COLOR), + ("fusion_model_b", 0, AOT_CIRCLE_COLOR), + ("fusion_model_a", 1, AOT_CIRCLE_COLOR), + ("fusion_model_b", 1, AOT_CIRCLE_COLOR), + ("fusion_model_a", 2, EXECUTE_COLOR), + ("fusion_model_b", 2, EXECUTE_COLOR), + + ] + + super().__init__(vertices, edges, layout, color_sequence, labels=labels, **kwargs) + + diff --git a/website/src/manim-visualizations/manim.cfg b/website/src/manim-visualizations/manim.cfg new file mode 100644 index 00000000000..0d5a6590589 --- /dev/null +++ b/website/src/manim-visualizations/manim.cfg @@ -0,0 +1,9 @@ +[CLI] +background_color = "#FFFFFF" + + +frame_width = 12.5 +frame_height = 6 + +pixel_width = 1400 +pixel_height = 240 #550 for the larger one diff --git a/website/src/manim-visualizations/requirements.txt b/website/src/manim-visualizations/requirements.txt new file mode 100644 index 00000000000..53e2d17e99b --- /dev/null +++ b/website/src/manim-visualizations/requirements.txt @@ -0,0 +1 @@ +manim>=0.19.0 \ No newline at end of file diff --git a/website/static/img/fusion/CoreJitRun.mp4 b/website/static/img/fusion/CoreJitRun.mp4 new file mode 100644 index 00000000000..dfe72fb7497 Binary files /dev/null and b/website/static/img/fusion/CoreJitRun.mp4 differ diff --git a/website/static/img/fusion/FusionAotRun.mp4 b/website/static/img/fusion/FusionAotRun.mp4 new file mode 100644 index 00000000000..103ef73b36a Binary files /dev/null and b/website/static/img/fusion/FusionAotRun.mp4 differ diff --git a/website/static/img/fusion/FusionJitCompileUnsafe.mp4 b/website/static/img/fusion/FusionJitCompileUnsafe.mp4 new file mode 100644 index 00000000000..07038af3aa3 Binary files /dev/null and b/website/static/img/fusion/FusionJitCompileUnsafe.mp4 differ diff --git a/website/static/img/fusion/FusionJitRunUnsafe.mp4 b/website/static/img/fusion/FusionJitRunUnsafe.mp4 new file mode 100644 index 00000000000..0891da53f76 Binary files /dev/null and b/website/static/img/fusion/FusionJitRunUnsafe.mp4 differ diff --git a/website/static/img/fusion/FusionJitRunUnsafeComplexDag.mp4 b/website/static/img/fusion/FusionJitRunUnsafeComplexDag.mp4 new file mode 100644 index 00000000000..7915567ad18 Binary files /dev/null and b/website/static/img/fusion/FusionJitRunUnsafeComplexDag.mp4 differ diff --git a/website/static/img/fusion/annotated_steps.png b/website/static/img/fusion/annotated_steps.png new file mode 100644 index 00000000000..608f21903d8 Binary files /dev/null and b/website/static/img/fusion/annotated_steps.png differ diff --git a/website/static/img/fusion/aot-compilation.png b/website/static/img/fusion/aot-compilation.png deleted file mode 100644 index 6893bf91c90..00000000000 Binary files a/website/static/img/fusion/aot-compilation.png and /dev/null differ diff --git a/website/static/img/fusion/introspection-mixed.png b/website/static/img/fusion/introspection-mixed.png deleted file mode 100644 index 1c30544808e..00000000000 Binary files a/website/static/img/fusion/introspection-mixed.png and /dev/null differ diff --git a/website/static/img/fusion/introspection-normal.png b/website/static/img/fusion/introspection-normal.png deleted file mode 100644 index ee3d5c7a22c..00000000000 Binary files a/website/static/img/fusion/introspection-normal.png and /dev/null differ diff --git a/website/static/img/fusion/jit-compilation.png b/website/static/img/fusion/jit-compilation.png deleted file mode 100644 index b32dfa1de6e..00000000000 Binary files a/website/static/img/fusion/jit-compilation.png and /dev/null differ