From b73167a795f126d2611b2438a4a2697e46a093c8 Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Tue, 15 Jul 2025 03:48:10 -0700 Subject: [PATCH 1/3] Document `excludeMetaHeaderRegex` kafka pubsub configuration Signed-off-by: Patrick Assuied --- .../supported-pubsub/setup-apache-kafka.md | 30 ++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 3a38b5bfa2c..09a7b297cd6 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -129,6 +129,7 @@ spec: | sessionTimeout | N | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer is removed and initiates a rebalance. Defaults to "10s". | `"20s"` | | consumerGroupRebalanceStrategy | N | The strategy to use for consumer group rebalancing. Supported values: `range`, `sticky`, `roundrobin`. Default is `range` | `"sticky"` | | escapeHeaders | N | Enables URL escaping of the message header values received by the consumer. Allows receiving content with special characters that are usually not allowed in HTTP headers. Default is `false`. | `true` | +| excludeHeaderMetaRegex | N | A regular expression to exclude keys from being converted from headers to metadata when consuming messages and from metadata to headers when publishing messages. This capability avoids unwanted downstream side effects for topic consumers. | '"^valueSchemaType$"` The `secretKeyRef` above is referencing a [kubernetes secrets store]({{< ref kubernetes-secret-store.md >}}) to access the tls information. Visit [here]({{< ref setup-secret-store.md >}}) to learn more about how to configure a secret store component. @@ -680,7 +681,34 @@ app.include_router(router) {{% /codetab %}} -{{< /tabs >}} +{{< /tabs >}} + +### Avoiding downstream side effects when publishing messages requiring custom metadata +Dapr allows customizing the publishing behavior by setting custom publish metadata. + +For instance, to publish in avro format, it is required to set `valueSchemaType=Avro` and `rawPayload=true` metadata. + +However, by default these metadata items get converted to Kafka headers and published along with the message. This default behavior is very helpful for instance to forward tracing headers across a chain of publishers/consumers. + +In certain scenario, however, it has unwanted side effects. +Let's assume you consume an Avro message using Dapr with the headers above.If this message cannot be consumed successfully and sent to a dead letter topic, the `valueSchemaType=Avro` and `rawPayload=true` will be automatically carried forward when publishing to the dead letter topic, requiring the set up of a schema associated with this topic. In many scenarios, it is preferable to publish dead letter messages in JSON only. + +To avoid this behavior, the kafka-pubsub component can be configured to exclude certain metadata keys from being converted to/from headers. +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: kafka-pubsub-exclude-metadaa + type: pubsub.kafka + version: v1 + metadata: + - name: brokers # Required. Kafka broker connection setting + value: "dapr-kafka.myapp.svc.cluster.local:9092" + - name: authType # Required. + value: "none" + - name: excludeMetaHeaderRegex + value: "^valueSchemaType$" # Optional. Excludes both `valueSchemaType` header from being published to headers and converted to metadata +``` ### Overriding default consumer group rebalancing In Kafka, rebalancing strategies determine how partitions are assigned to consumers within a consumer group. The default strategy is "range", but "roundrobin" and "sticky" are also available. From ac887f725d085a89330c194e35205a8de5b0bbe9 Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Tue, 15 Jul 2025 03:52:43 -0700 Subject: [PATCH 2/3] small adjustments Signed-off-by: Patrick Assuied --- .../supported-pubsub/setup-apache-kafka.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 09a7b297cd6..75b69bbe066 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -691,14 +691,14 @@ For instance, to publish in avro format, it is required to set `valueSchemaType= However, by default these metadata items get converted to Kafka headers and published along with the message. This default behavior is very helpful for instance to forward tracing headers across a chain of publishers/consumers. In certain scenario, however, it has unwanted side effects. -Let's assume you consume an Avro message using Dapr with the headers above.If this message cannot be consumed successfully and sent to a dead letter topic, the `valueSchemaType=Avro` and `rawPayload=true` will be automatically carried forward when publishing to the dead letter topic, requiring the set up of a schema associated with this topic. In many scenarios, it is preferable to publish dead letter messages in JSON only. +Let's assume you consume an Avro message using Dapr with the headers above.If this message cannot be consumed successfully and sent to a dead letter topic, `valueSchemaType=Avro` will be automatically carried forward when publishing to the dead letter topic, requiring the set up of a schema associated with this topic. In many scenarios, it is preferable to publish dead letter messages in JSON only. To avoid this behavior, the kafka-pubsub component can be configured to exclude certain metadata keys from being converted to/from headers. ```yaml apiVersion: dapr.io/v1alpha1 kind: Component metadata: - name: kafka-pubsub-exclude-metadaa + name: kafka-pubsub-exclude-metadata type: pubsub.kafka version: v1 metadata: @@ -707,7 +707,7 @@ metadata: - name: authType # Required. value: "none" - name: excludeMetaHeaderRegex - value: "^valueSchemaType$" # Optional. Excludes both `valueSchemaType` header from being published to headers and converted to metadata + value: "^valueSchemaType$" # Optional. Excludes `valueSchemaType` header from being published to headers and converted to metadata ``` ### Overriding default consumer group rebalancing From 706fb30acf1ec3fd52c00b8538b8df874d7f56c9 Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Tue, 15 Jul 2025 04:14:49 -0700 Subject: [PATCH 3/3] small adjustments. Signed-off-by: Patrick Assuied --- .../supported-pubsub/setup-apache-kafka.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 75b69bbe066..95040b09c76 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -686,12 +686,12 @@ app.include_router(router) ### Avoiding downstream side effects when publishing messages requiring custom metadata Dapr allows customizing the publishing behavior by setting custom publish metadata. -For instance, to publish in avro format, it is required to set `valueSchemaType=Avro` and `rawPayload=true` metadata. +For instance, to publish in avro format, it is required to set the `valueSchemaType=Avro` metadata. However, by default these metadata items get converted to Kafka headers and published along with the message. This default behavior is very helpful for instance to forward tracing headers across a chain of publishers/consumers. In certain scenario, however, it has unwanted side effects. -Let's assume you consume an Avro message using Dapr with the headers above.If this message cannot be consumed successfully and sent to a dead letter topic, `valueSchemaType=Avro` will be automatically carried forward when publishing to the dead letter topic, requiring the set up of a schema associated with this topic. In many scenarios, it is preferable to publish dead letter messages in JSON only. +Let's assume you consume an Avro message using Dapr with the headers above.If this message cannot be consumed successfully and configured to be sent to a dead letter topic, `valueSchemaType=Avro` will be automatically carried forward when publishing to the dead letter topic, requiring the set up of a schema associated with this topic. In many scenarios, it is preferable to publish dead letter messages in JSON only, as complying to a determined schema is not possible. To avoid this behavior, the kafka-pubsub component can be configured to exclude certain metadata keys from being converted to/from headers. ```yaml