[fix][io] Upgrade Debezium connector version to 2.6.1.Final

[mukesh154]

Fixes #22626

Motivation

To Upgrade debezium version to 2.6.1.Final for all source connectors.

It will address this comment as well.

Modifications

Upgraded debezium version from 1.9.7.Final to 2.6.1.Final all debezium source connectors which includes mysql, postgresql, mongodb, mssql & oracle.

Verifying this change

• Make sure that the change passes the CI checks.

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

According to https://debezium.io/ , latest stable Debezium release series is 2.7.x . Could we upgrade directly to 2.7.x instead?

[mukesh154]

From 2.6.2.Final onwards the postgres tester is not working. The consumer is expecting a message event just after the insertion into the table but the source is taking some time to process the event. I have tested this locally as well.
I think it is due to PostgreSQL offset flush race condition. More details are here:

1. Release Notes
2. DBZ-7816

[lhotari]

From 2.6.2.Final onwards the postgres tester is not working.

DBZ-7816 is fixed in 2.6.2.Final. Perhaps something needs to be adjusted in the Pulsar Debezium integration?

[lhotari]

Why is this change needed? I don't know Debezium very well, but it seems that "database history" and "schema history" are separate concepts.

[lhotari]

nvm. found the answer, DBZ-5043 Replace 'database history' with 'database schema history' and
DBZ-5043 Rename DatabaseHistory to SchemaHistory

[lhotari]

It might be more consistent naming if PulsarDatabaseHistory is renamed to PulsarSchemaHistory. For example, in Debezium, KafkaDatabaseHistory was renamed to KafkaSchemaHistory.

[mukesh154]

They renamed DatabaseHistory to SchemaHistory in 2.0 version of debezium. See here

[lhotari]

I'd recommend running tests for this PR in your own fork so that you get CI feedback without consuming apache/pulsar CI resources.

apache/pulsar pull requests should be first tested in your own fork since the apache/pulsar CI based on
GitHub Actions has constrained resources and quota. GitHub Actions provides separate quota for
pull requests that are executed in a forked repository.

1. Go to https://github.com/cognitree/pulsar/tree/master and ensure that your master branch is up to date
   with https://github.com/apache/pulsar
   Sync your fork if it's behind.
2. Open a pull request to your own fork. You can use this link to create the pull request in
   your own fork:
   Create PR in fork for running tests
3. Edit the description of the pull request [fix][io] Upgrade Debezium connector version to 2.6.1.Final #23078 and add the link to the PR that you opened to your own fork
   so that the reviewer can verify that tests pass in your own fork.
4. Ensure that tests pass in your own fork. Your own fork will be used to run the tests during the PR review
   and any changes made during the review. You as a PR author are responsible for following up on test failures.
   Please report any flaky tests as new issues at https://github.com/apache/pulsar/issues
   after checking that the flaky test isn't already reported.
5. When the PR is approved, it will be possible to restart the Pulsar CI workflow within apache/pulsar
   repository by adding a comment "/pulsarbot rerun-failure-checks" to the PR.
   An alternative for the PR approval is to add a ready-to-test label to the PR. This can be done
   by Apache Pulsar committers.
6. When tests pass on the apache/pulsar side, the PR can be merged by a Apache Pulsar Committer.

[mukesh154]

The PR checks passed on my fork for 2.6.1.Final. Check here.

[lhotari]

Since upgrading Debezium will be a breaking change. We need to document the upgrade notes for our users. What is the upgrade path and instructions for existing installations? Is it possible to upgrade an existing installation without loosing events etc.?

[mukesh154]

Here is the doc for the breaking changes.

[lhotari]

Superseded by #24712