[Bug] Topic does not exist in the remote cluster for non-paritioned topics

[saxena-gaurav]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

Broker version: 4.0.4

Issue Description

Non-Paritioned Topics are not getting created in the remote cluster if topic auto-creation is disabled. But partition topics work fine.

Error messages

Reproducing the issue

1. Create geo-replicating cluster
2. Disable topic auto-creation on the remote cluster.
3. Create non-parttion topic on the local cluster.
4. Send message to the topic on the local cluster.
5. Message will not replicate to the remote cluster.

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[zihenzzz]

i want to fix it

[poorbarcode]

@zihenzzz Could you show your solution first?

[zihenzzz]

@zihenzzz Could you show your solution first?

I think this is because Pulsar handles partitioned and non-partitioned topics differently: Partitioned topics: When replication is enabled, Pulsar actively creates the topic in the remote cluster without relying on the auto-creation feature. Non-partitioned topics: The replicators work like ordinary producers and rely on the auto-creation feature of the remote cluster. If the remote cluster has disabled auto-creation, the topic cannot be created, leading to replication failure.
The simplest solution - enable automatic creation on the remote cluster. The safest solution - manually create on both sides.

[poorbarcode]

@zihenzzz

The simplest solution - enable automatic creation on the remote cluster. The safest solution - manually create on both sides.

Good point, but the solution we should implement is to let replication work for the scenario where users are switching the policy of topic autop-creation

[zihenzzz]

@zihenzzz

The simplest solution - enable automatic creation on the remote cluster. The safest solution - manually create on both sides.

Good point, but the solution we should implement is to let replication work for the scenario where users are switching the policy of topic autop-creation

I have a general understanding. If possible, I can try to fix it

[poorbarcode]

@zihenzzz

I have a general understanding. If possible, I can try to fix it

Could you share the details of the implementation that you want to do? Because I feel this issue should fix through a PIP, it is not easy to fix

[zihenzzz]

@zihenzzz

I have a general understanding. If possible, I can try to fix it

Could you share the details of the implementation that you want to do? Because I feel this issue should fix through a PIP, it is not easy to fix

Add logic for actively creating non partitioned themes in GeoPersistentRepligator
When automatic creation is disabled for remote clusters:
Partition theme: Continue to use the existing creation mechanism
Non partitioned theme: Proactively created through newly added code paths
Is this okay? You can point out the problem and I will correct it

[poorbarcode]

Non partitioned theme: Proactively created through newly added code paths
Is this okay? You can point out the problem and I will correct it

Loos Good, how could you solve the issue that both clusters have different auto-creation as follows:

• Source cluster: auto-create a non-partitioned topic if it does not exist
• Target cluster: auto-create a partitioned topic with 2 partitions if it does not exist

[zihenzzz]

Non partitioned theme: Proactively created through newly added code paths
Is this okay? You can point out the problem and I will correct it

Loos Good, how could you solve the issue that both clusters have different auto-creation as follows:

• Source cluster: auto-create a non-partitioned topic if it does not exist
• Target cluster: auto-create a partitioned topic with 2 partitions if it does not exist

yes

[poorbarcode]

@zihenzzz

I have raised a proposal to solve the issue. Could you take a look? And we can work together for the following implementation

• [improve][pip] PIP-433: Optimize the conflicts of the replication and automatic creation mechanisms, including the automatic creation of topics and schemas #24485