Skip to content

Commit 5299b6d

Browse files
docs: Improve $parameters documentation with automatic propagation details (#69249)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 42e28e0 commit 5299b6d

File tree

2 files changed

+158
-6
lines changed
  • docs/platform/connector-development/config-based/advanced-topics
  • docusaurus/platform_versioned_docs/version-2.0/connector-development/config-based/advanced-topics

2 files changed

+158
-6
lines changed

docs/platform/connector-development/config-based/advanced-topics/parameters.md

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Schema:
1111
additionalProperties: true
1212
```
1313
14+
## Basic Usage
15+
1416
Example:
1517
1618
```yaml
@@ -21,7 +23,7 @@ outer:
2123
k2: v2
2224
```
2325
24-
This the example above, if both outer and inner are types with a "MyKey" field, both of them will evaluate to "MyValue".
26+
In this example, if both outer and inner are types with a "MyKey" field, both of them evaluate to "MyValue."
2527
2628
These parameters can be overwritten by subcomponents as a form of specialization:
2729
@@ -35,7 +37,7 @@ outer:
3537
k2: v2
3638
```
3739
38-
In this example, "outer.MyKey" will evaluate to "MyValue", and "inner.MyKey" will evaluate to "YourValue".
40+
In this example, "outer.MyKey" evaluates to "MyValue," and "inner.MyKey" evaluates to "YourValue."
3941
4042
The value can also be used for string interpolation:
4143
@@ -47,4 +49,78 @@ outer:
4749
k2: "MyKey is {{ parameters['MyKey'] }}"
4850
```
4951
50-
In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
52+
In this example, outer.inner.k2 evaluates to "MyKey is MyValue."
53+
54+
## Automatic Parameter Propagation
55+
56+
Parameters are automatically applied to component fields when those fields are not explicitly set. This happens recursively through nested components, allowing you to define parameters at a high level (like a stream) and have them automatically flow down to deeply nested components (like a requester inside a retriever).
57+
58+
### How It Works
59+
60+
When a component is processed:
61+
62+
1. Parameters from parent components are merged with the current component's parameters
63+
2. Each parameter key is checked against the component's fields
64+
3. If a field with that name exists and is not already set (or evaluates to false), the parameter value is assigned to that field
65+
4. The merged parameters are then passed down to all child components recursively
66+
67+
### Precedence Rules
68+
69+
- **Explicit values win**: If a field is explicitly set on a component, parameters do not override it
70+
- **Child parameters override parent parameters**: Parameters defined on a child component take precedence over those from parent components
71+
- **Exclusion rule**: When descending into a nested component, any parameter whose key matches the component's field name is temporarily excluded from propagation to avoid circular references
72+
73+
### Real-World Example
74+
75+
In the Stripe connector, multiple streams share the same base configuration but differ only in their API path. Here's how parameters enable this:
76+
77+
```yaml
78+
definitions:
79+
base_stream:
80+
type: DeclarativeStream
81+
retriever:
82+
$ref: "#/definitions/base_retriever"
83+
# ... other common configuration
84+
85+
base_retriever:
86+
type: SimpleRetriever
87+
requester:
88+
$ref: "#/definitions/base_requester"
89+
# ... other retriever configuration
90+
91+
base_requester:
92+
type: HttpRequester
93+
url_base: "https://api.stripe.com/v1"
94+
# Note: no 'path' field defined here
95+
96+
streams:
97+
shipping_rates:
98+
$ref: "#/definitions/base_stream"
99+
$parameters:
100+
path: shipping_rates
101+
name: shipping_rates
102+
schema_loader:
103+
# ... schema configuration
104+
105+
file_links:
106+
$ref: "#/definitions/base_stream"
107+
$parameters:
108+
path: file_links
109+
name: file_links
110+
schema_loader:
111+
# ... schema configuration
112+
```
113+
114+
In this example:
115+
116+
- Both streams reference the same `base_stream` definition
117+
- Each stream provides different `$parameters` values for `path` and `name`
118+
- These parameters automatically propagate down through the component hierarchy: stream → retriever → requester
119+
- The `path` parameter automatically sets the `requester.path` field, even though it's nested two levels deep
120+
- The `name` parameter sets the `stream.name` field
121+
122+
This pattern eliminates repetition and makes it easy to create multiple similar streams that differ only in a few key values.
123+
124+
### Technical Details
125+
126+
The parameter propagation mechanism is implemented in the `ManifestComponentTransformer.propagate_types_and_parameters` method in the CDK. For more details, see the [CDK source code](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py).

docusaurus/platform_versioned_docs/version-2.0/connector-development/config-based/advanced-topics/parameters.md

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Schema:
1111
additionalProperties: true
1212
```
1313
14+
## Basic Usage
15+
1416
Example:
1517
1618
```yaml
@@ -21,7 +23,7 @@ outer:
2123
k2: v2
2224
```
2325
24-
This the example above, if both outer and inner are types with a "MyKey" field, both of them will evaluate to "MyValue".
26+
In this example, if both outer and inner are types with a "MyKey" field, both of them evaluate to "MyValue."
2527
2628
These parameters can be overwritten by subcomponents as a form of specialization:
2729
@@ -35,7 +37,7 @@ outer:
3537
k2: v2
3638
```
3739
38-
In this example, "outer.MyKey" will evaluate to "MyValue", and "inner.MyKey" will evaluate to "YourValue".
40+
In this example, "outer.MyKey" evaluates to "MyValue," and "inner.MyKey" evaluates to "YourValue."
3941
4042
The value can also be used for string interpolation:
4143
@@ -47,4 +49,78 @@ outer:
4749
k2: "MyKey is {{ parameters['MyKey'] }}"
4850
```
4951
50-
In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
52+
In this example, outer.inner.k2 evaluates to "MyKey is MyValue."
53+
54+
## Automatic Parameter Propagation
55+
56+
Parameters are automatically applied to component fields when those fields are not explicitly set. This happens recursively through nested components, allowing you to define parameters at a high level (like a stream) and have them automatically flow down to deeply nested components (like a requester inside a retriever).
57+
58+
### How It Works
59+
60+
When a component is processed:
61+
62+
1. Parameters from parent components are merged with the current component's parameters
63+
2. Each parameter key is checked against the component's fields
64+
3. If a field with that name exists and is not already set (or evaluates to false), the parameter value is assigned to that field
65+
4. The merged parameters are then passed down to all child components recursively
66+
67+
### Precedence Rules
68+
69+
- **Explicit values win**: If a field is explicitly set on a component, parameters do not override it
70+
- **Child parameters override parent parameters**: Parameters defined on a child component take precedence over those from parent components
71+
- **Exclusion rule**: When descending into a nested component, any parameter whose key matches the component's field name is temporarily excluded from propagation to avoid circular references
72+
73+
### Real-World Example
74+
75+
In the Stripe connector, multiple streams share the same base configuration but differ only in their API path. Here's how parameters enable this:
76+
77+
```yaml
78+
definitions:
79+
base_stream:
80+
type: DeclarativeStream
81+
retriever:
82+
$ref: "#/definitions/base_retriever"
83+
# ... other common configuration
84+
85+
base_retriever:
86+
type: SimpleRetriever
87+
requester:
88+
$ref: "#/definitions/base_requester"
89+
# ... other retriever configuration
90+
91+
base_requester:
92+
type: HttpRequester
93+
url_base: "https://api.stripe.com/v1"
94+
# Note: no 'path' field defined here
95+
96+
streams:
97+
shipping_rates:
98+
$ref: "#/definitions/base_stream"
99+
$parameters:
100+
path: shipping_rates
101+
name: shipping_rates
102+
schema_loader:
103+
# ... schema configuration
104+
105+
file_links:
106+
$ref: "#/definitions/base_stream"
107+
$parameters:
108+
path: file_links
109+
name: file_links
110+
schema_loader:
111+
# ... schema configuration
112+
```
113+
114+
In this example:
115+
116+
- Both streams reference the same `base_stream` definition
117+
- Each stream provides different `$parameters` values for `path` and `name`
118+
- These parameters automatically propagate down through the component hierarchy: stream → retriever → requester
119+
- The `path` parameter automatically sets the `requester.path` field, even though it's nested two levels deep
120+
- The `name` parameter sets the `stream.name` field
121+
122+
This pattern eliminates repetition and makes it easy to create multiple similar streams that differ only in a few key values.
123+
124+
### Technical Details
125+
126+
The parameter propagation mechanism is implemented in the `ManifestComponentTransformer.propagate_types_and_parameters` method in the CDK. For more details, see the [CDK source code](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py).

0 commit comments

Comments
 (0)