[Python Functions] Close the configuration parity gaps with the Java runtime (master issue)

[david-streamlio]

Search before reporting

• I searched in the issues and found nothing similar.

This is a master issue. The individual gaps are tracked as the sub-issues listed below; this one holds the comparison that produced them. It is the Python counterpart to #26404, which tracks the same audit for the Go runtime.

Motivation

The Python function runtime is much closer to the Java reference than the Go runtime is, but it still drops a set of FunctionDetails fields. As with Go, the failures are silent: the configuration is accepted by pulsar-admin, stored by the worker, reported back by functions get, carried into the instance in the protobuf, and then never read.

I audited the three runtimes field by field against Function.proto, reading the consuming code rather than searching for field names — Function_pb2.py is generated and contains every field name, so a naive grep reports full support for everything.

Configuration	Java	Python	Go
receiverQueueSize	yes	yes	yes
subscriptionType / subscriptionName / subscriptionPosition	yes	yes	yes
timeoutMs	yes	yes	yes
retainOrdering / retainKeyOrdering	yes	yes	no (#26405)
cryptoSpec (consumer)	yes	yes	no (#26407)
schemaProperties (consumer)	yes	yes	no (#26407)
forwardSourceMessageProperty	yes	yes	no (#26408)
compressionType / batchBuilder	yes	yes	yes
userConfig / secretsMap / logTopic / autoAck	yes	yes	yes
batchingSpec	yes	no	no
retryDetails (deadLetterTopic, maxMessageRetries)	yes	no	no (#26406)
consumerProperties	yes	no	no
poolMessages	yes	no	no
messagePayloadProcessorSpec	yes	no	no
negativeAckRedeliveryDelayMs	yes	no	no (#26409)

(An earlier revision of this issue listed cleanupSubscription as a Python gap that Go honoured. Both halves were wrong: it is implemented by the worker in FunctionActioner, runtime-agnostically, so no instance runtime implements it and none needs to. The row has been removed.)

Producer maxPendingMessages and maxPendingMessagesAcrossPartitions are also unapplied on master, but they are already fixed by the open batching PR and are not tracked separately here.

Solution

Tracked individually:

• batchingSpec ignored, so every function has a fixed 10ms publish-latency floor — [Functions] Python function runtime hardcodes producer batching config, ignoring producerSpec.batchingSpec #26390 (PR open: [fix][fn] Honour producerSpec batching configuration in the Python function runtime #26392)
• retryDetails ignored, so no dead letter policy is created — [Python Functions] Python instance runtime silently ignores deadLetterTopic / maxMessageRetries #26397 (PR open: [fix][fn] Honour deadLetterTopic and maxMessageRetries in the Python function runtime #26400)
• consumerProperties, poolMessages and messagePayloadProcessorSpec ignored — [Python Functions] consumerProperties, poolMessages and messagePayloadProcessorSpec are ignored #26410
• negativeAckRedeliveryDelayMs ignored, so the 60s client default always applies — [Python Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26411

Two of the four already have fixes in review, so this issue is mostly a record of the remaining two plus the shape of the whole.

A note on how these should fail. The Go runtime refuses EFFECTIVELY_ONCE explicitly rather than ignoring it:

// pulsar-function-go/pf/instanceConf.go:137
panic("Go instance current not support EFFECTIVELY_ONCE processing guarantees.")

Every row marked "no" above fails the other way: accepted and dropped. Where a field is not going to be supported — messagePayloadProcessorSpec is the likely candidate here, being a Java-centric extension point — refusing it, or logging a warning at startup, is better than continuing to accept it silently. That is the same point raised in #26404 and is worth settling once for both runtimes.

Alternatives

Fixing these as one change was considered and rejected: they touch different parts of python_instance.py, two already have separate PRs in review, and cleanupSubscription may not belong in the instance at all — it may be the worker's delete path that needs it, which is a different discussion from the consumer-option fields.

Anything else?

The audit was done against origin/master. Java is treated as the reference implementation; I spot-checked the Java side for the rows that mattered (PulsarSource for retryDetails and the negative-ack delay, BatchingUtils for batchingSpec, JavaInstanceRunnable for the presence guards) rather than auditing it exhaustively.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[david-streamlio]

Three of the four sub-issues now have PRs, all green:

• [Functions] Python function runtime hardcodes producer batching config, ignoring producerSpec.batchingSpec #26390 (batchingSpec) → [fix][fn] Honour producerSpec batching configuration in the Python function runtime #26392
• [Python Functions] Python instance runtime silently ignores deadLetterTopic / maxMessageRetries #26397 (dead letter policy) → [fix][fn] Honour deadLetterTopic and maxMessageRetries in the Python function runtime #26400
• [Python Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26411 (negativeAckRedeliveryDelayMs) → [fix][fn] Honour negativeAckRedeliveryDelayMs in the Python function runtime #26413

Remaining: #26410, where only consumerProperties is actually implementable today — pool_messages is not in the Python client's subscribe() signature at all, and messagePayloadProcessorSpec has no Python equivalent. That issue may be worth narrowing to consumerProperties and closing the other two as unsupported, which would leave this master issue fully covered by open PRs.

@freeznet — flagging for the same reason as #26404.