Skip to content

Commit 9cea2f6

Browse files
stevhliustevhliu
committed
sequential
1 parent 44c521d commit 9cea2f6

File tree

2 files changed

+81
-147
lines changed

2 files changed

+81
-147
lines changed

‎docs/source/en/_toctree.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,8 @@
262262
title: Pipeline blocks
263263
- local: api/modular_diffusers/pipeline_states
264264
title: Pipeline states
265+
- local: api/modular_diffusers/pipeline_components
266+
title: Components and configs
265267
title: Modular Diffusers
266268
- isExpanded: false
267269
sections:

‎docs/source/en/modular_diffusers/sequential_pipeline_blocks.md

Lines changed: 79 additions & 147 deletions
Original file line numberDiff line numberDiff line change
@@ -12,178 +12,110 @@ specific language governing permissions and limitations under the License.
1212

1313
# SequentialPipelineBlocks
1414

15-
<Tip warning={true}>
15+
[`SequentialPipelineBlocks`] are a multi-block type that composes other [`PipelineBlocks`] together in a sequence. Data flows linearly from one block to the next using `intermediate_inputs` and `intermediate_outputs`. Each block in [`SequentialPipelineBlocks`] usually represents a step in the pipeline, and by combining them, you gradually build a pipeline.
1616

17-
🧪 **Experimental Feature**: Modular Diffusers is an experimental feature we are actively developing. The API may be subject to breaking changes.
17+
This guide shows you how to connect two blocks into a [`SequentialPipelineBlocks`].
1818

19-
</Tip>
19+
Create two [`PipelineBlocks`]. The first block, `InputBlock`, outputs a `batch_size` value and the second block, `ImageEncoderBlock` uses `batch_size` as `intermediate_inputs`.
2020

21-
`SequentialPipelineBlocks` is a subclass of `ModularPipelineBlocks`. Unlike `PipelineBlock`, it is a multi-block that composes other blocks together in sequence, creating modular workflows where data flows from one block to the next. It's one of the most common ways to build complex pipelines by combining simpler building blocks.
22-
23-
<Tip>
24-
25-
Other types of multi-blocks include [AutoPipelineBlocks](auto_pipeline_blocks.md) (for conditional block selection) and [LoopSequentialPipelineBlocks](loop_sequential_pipeline_blocks.md) (for iterative workflows). For information on creating individual blocks, see the [PipelineBlock guide](pipeline_block.md).
26-
27-
Additionally, like all `ModularPipelineBlocks`, `SequentialPipelineBlocks` are definitions/specifications, not runnable pipelines. You need to convert them into a `ModularPipeline` to actually execute them. For information on creating and running pipelines, see the [Modular Pipeline guide](modular_pipeline.md).
28-
29-
</Tip>
30-
31-
In this tutorial, we will focus on how to create `SequentialPipelineBlocks` and how blocks connect and work together.
32-
33-
The key insight is that blocks connect through their intermediate inputs and outputs - the "studs and anti-studs" we discussed in the [PipelineBlock guide](pipeline_block.md). When one block produces an intermediate output, it becomes available as an intermediate input for subsequent blocks.
34-
35-
Let's explore this through an example. We will use the same helper function from the PipelineBlock guide to create blocks.
21+
<hfoptions id="sequential">
22+
<hfoption id="InputBlock">
3623

3724
```py
3825
from diffusers.modular_pipelines import PipelineBlock, InputParam, OutputParam
39-
import torch
4026

41-
def make_block(inputs=[], intermediate_inputs=[], intermediate_outputs=[], block_fn=None, description=None):
42-
class TestBlock(PipelineBlock):
43-
model_name = "test"
44-
45-
@property
46-
def inputs(self):
47-
return inputs
48-
49-
@property
50-
def intermediate_inputs(self):
51-
return intermediate_inputs
52-
53-
@property
54-
def intermediate_outputs(self):
55-
return intermediate_outputs
56-
57-
@property
58-
def description(self):
59-
return description if description is not None else ""
60-
61-
def __call__(self, components, state):
62-
block_state = self.get_block_state(state)
63-
if block_fn is not None:
64-
block_state = block_fn(block_state, state)
65-
self.set_block_state(state, block_state)
66-
return components, state
67-
68-
return TestBlock
27+
class InputBlock(PipelineBlock):
28+
29+
@property
30+
def inputs(self):
31+
return [
32+
InputParam(name="prompt", type_hint=list, description="list of text prompts"),
33+
InputParam(name="num_images_per_prompt", type_hint=int, description="number of images per prompt"),
34+
]
35+
36+
@property
37+
def intermediate_inputs(self):
38+
return []
39+
40+
@property
41+
def intermediate_outputs(self):
42+
return [
43+
OutputParam(name="batch_size", description="calculated batch size"),
44+
]
45+
46+
@property
47+
def description(self):
48+
return "A block that determines batch_size based on the number of prompts and num_images_per_prompt argument."
49+
50+
def __call__(self, components, state):
51+
block_state = self.get_block_state(state)
52+
batch_size = len(block_state.prompt)
53+
block_state.batch_size = batch_size * block_state.num_images_per_prompt
54+
self.set_block_state(state, block_state)
55+
return components, state
6956
```
7057

71-
Let's create a block that produces `batch_size`, which we'll call "input_block":
58+
</hfoption>
59+
<hfoption id="ImageEncoderBlock">
7260

7361
```py
74-
def input_block_fn(block_state, pipeline_state):
75-
76-
batch_size = len(block_state.prompt)
77-
block_state.batch_size = batch_size * block_state.num_images_per_prompt
78-
79-
return block_state
80-
81-
input_block_cls = make_block(
82-
inputs=[
83-
InputParam(name="prompt", type_hint=list, description="list of text prompts"),
84-
InputParam(name="num_images_per_prompt", type_hint=int, description="number of images per prompt")
85-
],
86-
intermediate_outputs=[
87-
OutputParam(name="batch_size", description="calculated batch size")
88-
],
89-
block_fn=input_block_fn,
90-
description="A block that determines batch_size based on the number of prompts and num_images_per_prompt argument."
91-
)
92-
input_block = input_block_cls()
62+
import torch
63+
from diffusers.modular_pipelines import PipelineBlock, InputParam, OutputParam
64+
65+
class ImageEncoderBlock(PipelineBlock):
66+
67+
@property
68+
def inputs(self):
69+
return [
70+
InputParam(name="image", type_hint="PIL.Image", description="raw input image to process"),
71+
]
72+
73+
@property
74+
def intermediate_inputs(self):
75+
return [
76+
InputParam(name="batch_size", type_hint=int),
77+
]
78+
79+
@property
80+
def intermediate_outputs(self):
81+
return [
82+
OutputParam(name="image_latents", description="latents representing the image"),
83+
]
84+
85+
@property
86+
def description(self):
87+
return "Encode raw image into its latent presentation"
88+
89+
def __call__(self, components, state):
90+
block_state = self.get_block_state(state)
91+
# Simulate processing the image
92+
block_state.image = torch.randn(1, 3, 512, 512)
93+
block_state.batch_size = block_state.batch_size * 2
94+
block_state.image_latents = torch.randn(1, 4, 64, 64)
95+
self.set_block_state(state, block_state)
96+
return components, state
9397
```
9498

95-
Now let's create a second block that uses the `batch_size` from the first block:
99+
</hfoption>
100+
</hfoptions>
96101

97-
```py
98-
def image_encoder_block_fn(block_state, pipeline_state):
99-
# Simulate processing the image
100-
block_state.image = torch.randn(1, 3, 512, 512)
101-
block_state.batch_size = block_state.batch_size * 2
102-
block_state.image_latents = torch.randn(1, 4, 64, 64)
103-
return block_state
104-
105-
image_encoder_block_cls = make_block(
106-
inputs=[
107-
InputParam(name="image", type_hint="PIL.Image", description="raw input image to process")
108-
],
109-
intermediate_inputs=[
110-
InputParam(name="batch_size", type_hint=int)
111-
],
112-
intermediate_outputs=[
113-
OutputParam(name="image_latents", description="latents representing the image")
114-
],
115-
block_fn=image_encoder_block_fn,
116-
description="Encode raw image into its latent presentation"
117-
)
118-
image_encoder_block = image_encoder_block_cls()
119-
```
102+
Connect the two blocks by defining an [`InsertableDict`] to map the block names to the block instances. Blocks are executed in the order they're registered in `blocks_dict`.
120103

121-
Now let's connect these blocks to create a `SequentialPipelineBlocks`:
104+
Use [`~SequentialPipelineBlocks.from_blocks_dict`] to create a [`SequentialPipelineBlocks`].
122105

123106
```py
124107
from diffusers.modular_pipelines import SequentialPipelineBlocks, InsertableDict
125108

126-
# Define a dict mapping block names to block instances
127109
blocks_dict = InsertableDict()
128110
blocks_dict["input"] = input_block
129111
blocks_dict["image_encoder"] = image_encoder_block
130112

131-
# Create the SequentialPipelineBlocks
132113
blocks = SequentialPipelineBlocks.from_blocks_dict(blocks_dict)
133114
```
134115

135-
Now you have a `SequentialPipelineBlocks` with 2 blocks:
116+
Inspect the sub-blocks in [`SequentialPipelineBlocks`] by calling `blocks`, and for more details about the inputs and outputs, access the `docs` attribute.
136117

137118
```py
138-
>>> blocks
139-
SequentialPipelineBlocks(
140-
Class: ModularPipelineBlocks
141-
142-
Description:
143-
144-
145-
Sub-Blocks:
146-
[0] input (TestBlock)
147-
Description: A block that determines batch_size based on the number of prompts and num_images_per_prompt argument.
148-
149-
[1] image_encoder (TestBlock)
150-
Description: Encode raw image into its latent presentation
151-
152-
)
153-
```
154-
155-
When you inspect `blocks.doc`, you can see that `batch_size` is not listed as an input. The pipeline automatically detects that the `input_block` can produce `batch_size` for the `image_encoder_block`, so it doesn't ask the user to provide it.
156-
157-
```py
158-
>>> print(blocks.doc)
159-
class SequentialPipelineBlocks
160-
161-
Inputs:
162-
163-
prompt (`None`, *optional*):
164-
165-
num_images_per_prompt (`None`, *optional*):
166-
167-
image (`PIL.Image`, *optional*):
168-
raw input image to process
169-
170-
Outputs:
171-
172-
batch_size (`None`):
173-
174-
image_latents (`None`):
175-
latents representing the image
176-
```
177-
178-
At runtime, you have data flow like this:
179-
180-
![Data Flow Diagram](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/Editor%20_%20Mermaid%20Chart-2025-06-30-092631.png)
181-
182-
**How SequentialPipelineBlocks Works:**
183-
184-
1. Blocks are executed in the order they're registered in the `blocks_dict`
185-
2. Outputs from one block become available as intermediate inputs to all subsequent blocks
186-
3. The pipeline automatically figures out which values need to be provided by the user and which will be generated by previous blocks
187-
4. Each block maintains its own behavior and operates through its defined interface, while collectively these interfaces determine what the entire pipeline accepts and produces
188-
189-
What happens within each block follows the same pattern we described earlier: each block gets its own `block_state` with the relevant inputs and intermediate inputs, performs its computation, and updates the pipeline state with its intermediate outputs.
119+
print(blocks)
120+
print(blocks.doc)
121+
```

0 commit comments

Comments
 (0)