Debezium HTTP-based integration

[yanivbh1]

Summary

Enable connectivity between Debezium to Memphis to enable CDC use cases.

Context

Multiple streaming pipelines require support in CDC for different databases. Debezium gathers the most popular ones like MySQL, Postgre, and more in a single framework

Value

Faster creation of CDC-based streaming pipelines

Persona(s)

• DevOps
• Data Engineers
• Developers

Asked by

• Yehuda Korotkin

Released on v1.1 and can be found on both the integration center and detailed here as well - https://memphis.dev/blog/part-1-integrating-debezium-server-and-memphis-dev-for-streaming-change-data-capture-cdc-events/

[yanivbh1]

debezium/debezium-server#2 (comment)

[rnowling-memphis]

@yanivbh1 Can we close this as completed and notify the person who requested the feature?

[yanivbh1]

@rnowling-memphis Yes!
I will notify Yehuda personally, but would be great to mention the users who added upvotes.
@idanasulinmemphis can you please add it to your list for the upcoming release notes?

[yanivbh1]

Also adding @avrham @avrhamNeeman ,
Let’s add debezium to the integration center with a quick guide to make sure our users aware to it.

[avrhamNeeman]

@yanivbh1 @rnowling-memphis We're ready with a graphic of the Debezium integration. Please provide the relevant content.

[yanivbh1]

@avrhamNeeman
I realized that we cannot unify all the different debezium abilities in one box, and it needs to change to
"Postgres and Debezium"

Box Description -
Debezium is one of the most popular frameworks for collecting "Change Data Capture (CDC)" events from various databases and can now be easily integrated with Memphis.dev for collecting CDC events from various databases.

Steps within -
A full guiding tutorial can be found here (https://memphis.dev/blog/part-1-integrating-debezium-server-and-memphis-dev-for-streaming-change-data-capture-cdc-events/)

Step 1: Create an application-type Memphis user to be used by Debezium
Link to the users' page

Step 2: Setup Debezium
Required Debezium configuration (normally stored in the application.properties file).

debezium.sink.type=http
debezium.sink.http.url=http://<Memphis REST Gateway URL>:4444/stations/todo-cdc-events/produce/single
debezium.sink.http.time-out.ms=500
debezium.sink.http.retries=3
debezium.sink.http.authentication.type=jwt
debezium.sink.http.authentication.jwt.username=<Memphis Application-type username>
debezium.sink.http.authentication.jwt.password=<Memphis Application-type password>
debezium.sink.http.authentication.jwt.url=http://<Memphis REST Gateway URL>:4444/
debezium.format.key=json
debezium.format.value=json
quarkus.log.console.json=false

In case Debezium is not installed yet, here is a quick Dockerfile to start one (Don't forget to attach the config file within the container)

FROM debian:bullseye-slim

RUN apt update && apt upgrade -y && apt install -y openjdk-11-jdk-headless wget git curl && rm -rf /var/cache/apt/*

WORKDIR /
RUN git clone https://github.com/debezium/debezium
WORKDIR /debezium
RUN ./mvnw clean install -DskipITs -DskipTests
WORKDIR /
RUN git clone https://github.com/debezium/debezium-server debezium-server-build
WORKDIR /debezium-server-build
RUN ./mvnw package -DskipITs -DskipTests -Passembly
RUN tar -xzvf debezium-server-dist/target/debezium-server-dist-*.tar.gz -C /
WORKDIR /debezium-server
RUN mkdir data

CMD ./run.sh

[rnowling-memphis]

Edited the above to add the description for setting up the WAL level.