You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/platform/connector-development/config-based/advanced-topics/parameters.md
+79-3Lines changed: 79 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,8 @@ Schema:
11
11
additionalProperties: true
12
12
```
13
13
14
+
## Basic Usage
15
+
14
16
Example:
15
17
16
18
```yaml
@@ -21,7 +23,7 @@ outer:
21
23
k2: v2
22
24
```
23
25
24
-
This the example above, if both outer and inner are types with a "MyKey" field, both of them will evaluate to "MyValue".
26
+
In this example, if both outer and inner are types with a "MyKey" field, both of them evaluate to "MyValue."
25
27
26
28
These parameters can be overwritten by subcomponents as a form of specialization:
27
29
@@ -35,7 +37,7 @@ outer:
35
37
k2: v2
36
38
```
37
39
38
-
In this example, "outer.MyKey" will evaluate to "MyValue", and "inner.MyKey" will evaluate to "YourValue".
40
+
In this example, "outer.MyKey" evaluates to "MyValue," and "inner.MyKey" evaluates to "YourValue."
39
41
40
42
The value can also be used for string interpolation:
41
43
@@ -47,4 +49,78 @@ outer:
47
49
k2: "MyKey is {{ parameters['MyKey'] }}"
48
50
```
49
51
50
-
In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
52
+
In this example, outer.inner.k2 evaluates to "MyKey is MyValue."
53
+
54
+
## Automatic Parameter Propagation
55
+
56
+
Parameters are automatically applied to component fields when those fields are not explicitly set. This happens recursively through nested components, allowing you to define parameters at a high level (like a stream) and have them automatically flow down to deeply nested components (like a requester inside a retriever).
57
+
58
+
### How It Works
59
+
60
+
When a component is processed:
61
+
62
+
1. Parameters from parent components are merged with the current component's parameters
63
+
2. Each parameter key is checked against the component's fields
64
+
3. If a field with that name exists and is not already set (or evaluates to false), the parameter value is assigned to that field
65
+
4. The merged parameters are then passed down to all child components recursively
66
+
67
+
### Precedence Rules
68
+
69
+
- **Explicit values win**: If a field is explicitly set on a component, parameters do not override it
70
+
- **Child parameters override parent parameters**: Parameters defined on a child component take precedence over those from parent components
71
+
- **Exclusion rule**: When descending into a nested component, any parameter whose key matches the component's field name is temporarily excluded from propagation to avoid circular references
72
+
73
+
### Real-World Example
74
+
75
+
In the Stripe connector, multiple streams share the same base configuration but differ only in their API path. Here's how parameters enable this:
76
+
77
+
```yaml
78
+
definitions:
79
+
base_stream:
80
+
type: DeclarativeStream
81
+
retriever:
82
+
$ref: "#/definitions/base_retriever"
83
+
# ... other common configuration
84
+
85
+
base_retriever:
86
+
type: SimpleRetriever
87
+
requester:
88
+
$ref: "#/definitions/base_requester"
89
+
# ... other retriever configuration
90
+
91
+
base_requester:
92
+
type: HttpRequester
93
+
url_base: "https://api.stripe.com/v1"
94
+
# Note: no 'path' field defined here
95
+
96
+
streams:
97
+
shipping_rates:
98
+
$ref: "#/definitions/base_stream"
99
+
$parameters:
100
+
path: shipping_rates
101
+
name: shipping_rates
102
+
schema_loader:
103
+
# ... schema configuration
104
+
105
+
file_links:
106
+
$ref: "#/definitions/base_stream"
107
+
$parameters:
108
+
path: file_links
109
+
name: file_links
110
+
schema_loader:
111
+
# ... schema configuration
112
+
```
113
+
114
+
In this example:
115
+
116
+
- Both streams reference the same `base_stream` definition
117
+
- Each stream provides different `$parameters` values for `path` and `name`
118
+
- These parameters automatically propagate down through the component hierarchy: stream → retriever → requester
119
+
- The `path` parameter automatically sets the `requester.path` field, even though it's nested two levels deep
120
+
- The `name` parameter sets the `stream.name` field
121
+
122
+
This pattern eliminates repetition and makes it easy to create multiple similar streams that differ only in a few key values.
123
+
124
+
### Technical Details
125
+
126
+
The parameter propagation mechanism is implemented in the `ManifestComponentTransformer.propagate_types_and_parameters` method in the CDK. For more details, see the [CDK source code](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py).
Copy file name to clipboardExpand all lines: docusaurus/platform_versioned_docs/version-2.0/connector-development/config-based/advanced-topics/parameters.md
+79-3Lines changed: 79 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,8 @@ Schema:
11
11
additionalProperties: true
12
12
```
13
13
14
+
## Basic Usage
15
+
14
16
Example:
15
17
16
18
```yaml
@@ -21,7 +23,7 @@ outer:
21
23
k2: v2
22
24
```
23
25
24
-
This the example above, if both outer and inner are types with a "MyKey" field, both of them will evaluate to "MyValue".
26
+
In this example, if both outer and inner are types with a "MyKey" field, both of them evaluate to "MyValue."
25
27
26
28
These parameters can be overwritten by subcomponents as a form of specialization:
27
29
@@ -35,7 +37,7 @@ outer:
35
37
k2: v2
36
38
```
37
39
38
-
In this example, "outer.MyKey" will evaluate to "MyValue", and "inner.MyKey" will evaluate to "YourValue".
40
+
In this example, "outer.MyKey" evaluates to "MyValue," and "inner.MyKey" evaluates to "YourValue."
39
41
40
42
The value can also be used for string interpolation:
41
43
@@ -47,4 +49,78 @@ outer:
47
49
k2: "MyKey is {{ parameters['MyKey'] }}"
48
50
```
49
51
50
-
In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
52
+
In this example, outer.inner.k2 evaluates to "MyKey is MyValue."
53
+
54
+
## Automatic Parameter Propagation
55
+
56
+
Parameters are automatically applied to component fields when those fields are not explicitly set. This happens recursively through nested components, allowing you to define parameters at a high level (like a stream) and have them automatically flow down to deeply nested components (like a requester inside a retriever).
57
+
58
+
### How It Works
59
+
60
+
When a component is processed:
61
+
62
+
1. Parameters from parent components are merged with the current component's parameters
63
+
2. Each parameter key is checked against the component's fields
64
+
3. If a field with that name exists and is not already set (or evaluates to false), the parameter value is assigned to that field
65
+
4. The merged parameters are then passed down to all child components recursively
66
+
67
+
### Precedence Rules
68
+
69
+
- **Explicit values win**: If a field is explicitly set on a component, parameters do not override it
70
+
- **Child parameters override parent parameters**: Parameters defined on a child component take precedence over those from parent components
71
+
- **Exclusion rule**: When descending into a nested component, any parameter whose key matches the component's field name is temporarily excluded from propagation to avoid circular references
72
+
73
+
### Real-World Example
74
+
75
+
In the Stripe connector, multiple streams share the same base configuration but differ only in their API path. Here's how parameters enable this:
76
+
77
+
```yaml
78
+
definitions:
79
+
base_stream:
80
+
type: DeclarativeStream
81
+
retriever:
82
+
$ref: "#/definitions/base_retriever"
83
+
# ... other common configuration
84
+
85
+
base_retriever:
86
+
type: SimpleRetriever
87
+
requester:
88
+
$ref: "#/definitions/base_requester"
89
+
# ... other retriever configuration
90
+
91
+
base_requester:
92
+
type: HttpRequester
93
+
url_base: "https://api.stripe.com/v1"
94
+
# Note: no 'path' field defined here
95
+
96
+
streams:
97
+
shipping_rates:
98
+
$ref: "#/definitions/base_stream"
99
+
$parameters:
100
+
path: shipping_rates
101
+
name: shipping_rates
102
+
schema_loader:
103
+
# ... schema configuration
104
+
105
+
file_links:
106
+
$ref: "#/definitions/base_stream"
107
+
$parameters:
108
+
path: file_links
109
+
name: file_links
110
+
schema_loader:
111
+
# ... schema configuration
112
+
```
113
+
114
+
In this example:
115
+
116
+
- Both streams reference the same `base_stream` definition
117
+
- Each stream provides different `$parameters` values for `path` and `name`
118
+
- These parameters automatically propagate down through the component hierarchy: stream → retriever → requester
119
+
- The `path` parameter automatically sets the `requester.path` field, even though it's nested two levels deep
120
+
- The `name` parameter sets the `stream.name` field
121
+
122
+
This pattern eliminates repetition and makes it easy to create multiple similar streams that differ only in a few key values.
123
+
124
+
### Technical Details
125
+
126
+
The parameter propagation mechanism is implemented in the `ManifestComponentTransformer.propagate_types_and_parameters` method in the CDK. For more details, see the [CDK source code](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py).
0 commit comments