Skip to content

mysql clickpipe: binlog faq #3837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 23, 2025
Merged

mysql clickpipe: binlog faq #3837

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0d4f009
mysql clickpipe: binlog faq
serprex May 22, 2025
edea5d5
lint
serprex May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions docs/integrations/data-ingestion/clickpipes/mysql/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,15 @@
# ClickPipes for MySQL FAQ

### Does the MySQL ClickPipe support MariaDB? {#does-the-clickpipe-support-mariadb}
Yes, the MySQL ClickPipe supports MariaDB 10.0 and above. The configuration for it is very similar to MySQL, with the GTID behaviour being enabled by default.
Yes, the MySQL ClickPipe supports MariaDB 10.0 and above. The configuration for it is very similar to MySQL, using GTID replication by default.

Check notice on line 12 in docs/integrations/data-ingestion/clickpipes/mysql/faq.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Contractions 

Suggestion: Use 'it's' instead of 'it is'.

Check notice on line 12 in docs/integrations/data-ingestion/clickpipes/mysql/faq.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Uppercase 

Suggestion: Instead of uppercase for 'GTID', use lowercase or backticks (`) if possible. Otherwise, ask a Technical Writer to add this word or acronym to the rule's exception list.

### Does the MySQL ClickPipe support Planetscale, Vitess? {#does-the-clickpipe-support-planetscale-vitess}
Currently, we support only standard MySQL. Since PlanetScale is built on Vitess, integrating with Vitess's VStream API and handling VGtids (Vitess Global Transaction IDs) to track incremental changes is required. This differs from how CDC operates in native MySQL. Adding support for this functionality is being actively worked on.
### Does the MySQL ClickPipe support PlanetScale, Vitess, or TiDB? {#does-the-clickpipe-support-planetscale-vitess}

Check failure on line 14 in docs/integrations/data-ingestion/clickpipes/mysql/faq.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Headings 

'Does the MySQL ClickPipe support PlanetScale, Vitess, or TiDB? {#does-the-clickpipe-support-planetscale-vitess}' should use sentence-style capitalization.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sentence casing this is inconsistent with rest of doc (also I tried appeasing it, maybe TiDB is bothering it?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serprex I added MySQL, ClickPipe, PlanetScale, Vitess and TiDB as exceptions and it still wasn't happy :) so we'll ignore it

No, these do not support MySQL's binlog API.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did an evaluation of these alternative's CDC story. Vitess CDC lacks authentication (ie, you connect to grpc endpoint & have complete access). As a result PlanetScale made their own CDC protocol which we would have to implement based on their airbyte connector as a reference implementation. TiDB deprecated support for binlog API, instead implementing TiCDC

This fractured landscape makes implementing a connector for these services less appetizing, to the point that we are not actively working on any of them


### How is replication managed? {#how-is-replication-managed}
We support both `GTID` & `FilePos` replication. Unlike Postgres there is no slot to manage offset. Instead, you must configure your MySQL server to have a sufficient binlog retention period. If our offset into the binlog becomes invalidated *(eg, mirror paused too long, or database failover occurs while using `FilePos` replication)* then you will need to resync the pipe. Make sure to optimize materialized views depending on destination tables, as inefficient queries can slow down ingestion to fall behind the retention period.

It's also possible for an inactive database to rotate the log file without allowing ClickPipes to progress to a more recent offset. You may need to setup a heartbeat table with regularly scheduled updates.

### Why am I getting a TLS certificate validation error when connecting to MySQL? {#tls-certificate-validation-error}
If you see an error like `failed to verify certificate: x509: certificate is not valid for any names`, this occurs when the SSL/TLS certificate on your MySQL server doesn't include the connecting hostname (e.g., EC2 instance DNS name) in its list of valid names. ClickPipes enables TLS by default to provide secure encrypted connections.
Expand Down
Loading