Skip to content

change FillZeroLog config name and add to canal #1053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 17, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ import (
"github.com/go-mysql-org/go-mysql/replication"
"os"
)
// Create a binlog syncer with a unique server id, the server id must be different from other MySQL's.
// Create a binlog syncer with a unique server id, the server id must be different from other MySQL's.
// flavor is mysql or mariadb
cfg := replication.BinlogSyncerConfig {
ServerID: 100,
Expand Down Expand Up @@ -137,7 +137,7 @@ Query: DROP TABLE IF EXISTS `test_replication` /* generated by server */

MariaDB 11.4+ introduced an optimization where events written through transaction or statement cache have `LogPos=0` so they can be copied directly to the binlog without computing the real end position. This optimization improves performance but makes position tracking unreliable for replication clients that need to track LogPos of events inside transactions.

To address this, a `MariaDBDynamicLogPos` configuration option is available:
To address this, a `FillZeroLogPos` configuration option is available:

```go
cfg := replication.BinlogSyncerConfig {
Expand All @@ -148,24 +148,24 @@ cfg := replication.BinlogSyncerConfig {
User: "root",
Password: "",
// Enable dynamic LogPos calculation for MariaDB 11.4+
MariaDBDynamicLogPos: true,
FillZeroLogPos: true,
}
```

**Behavior:**
- When `MariaDBDynamicLogPos` is `true` and flavor is `mariadb`, the library automatically:
- When `FillZeroLogPos` is `true` and flavor is `mariadb`, the library automatically:
- Adds `BINLOG_SEND_ANNOTATE_ROWS_EVENT` flag to binlog dump commands. This ensures correct position tracking by making the server send `ANNOTATE_ROWS_EVENT` events which are needed for accurate position calculation.
- Calculates LogPos dynamically for events with `LogPos=0` that are not artificial.
- Calculates LogPos dynamically for events with `LogPos=0` that are not artificial.
- Only works with MariaDB flavor; has no effect with MySQL.
- Should be set to `true` if tracking of LogPos inside transactions is required.

## Canal

Canal is a package that can sync your MySQL into everywhere, like Redis, Elasticsearch.

First, canal will dump your MySQL data then sync changed data using binlog incrementally.
First, canal will dump your MySQL data then sync changed data using binlog incrementally.

You must use ROW format for binlog, full binlog row image is preferred, because we may meet some errors when primary key changed in update for minimal or noblob row image.
You must use ROW format for binlog, full binlog row image is preferred, because we may meet some errors when primary key changed in update for minimal or noblob row image.

A simple example:

Expand Down Expand Up @@ -214,9 +214,9 @@ You can see [go-mysql-elasticsearch](https://github.com/go-mysql-org/go-mysql-el

## Client

Client package supports a simple MySQL connection driver which you can use it to communicate with MySQL server.
Client package supports a simple MySQL connection driver which you can use it to communicate with MySQL server.

For an example see [`example_client_test.go`](client/example_client_test.go). You can run this testable example with
For an example see [`example_client_test.go`](client/example_client_test.go). You can run this testable example with
`go test -v ./client -run Example`.

Tested MySQL versions for the client include:
Expand Down Expand Up @@ -263,7 +263,7 @@ conn.Execute() / conn.Begin() / etc...

## Server

Server package supplies a framework to implement a simple MySQL server which can handle the packets from the MySQL client.
Server package supplies a framework to implement a simple MySQL server which can handle the packets from the MySQL client.
You can use it to build your own MySQL proxy. The server connection is compatible with MySQL 5.5, 5.6, 5.7, and 8.0 versions,
so that most MySQL clients should be able to connect to the Server without modifications.

Expand Down Expand Up @@ -519,7 +519,7 @@ We pass all tests in https://github.com/bradfitz/go-sql-test using go-mysql driv

Logging uses [log/slog](https://pkg.go.dev/log/slog) and by default is sent to standard out.

For the old logging package `github.com/siddontang/go-log/log`, a converting package
For the old logging package `github.com/siddontang/go-log/log`, a converting package
`https://github.com/serprex/slog-siddontang` is available.
## How to migrate to this repo
To change the used package in your repo it's enough to add this `replace` directive to your `go.mod`:
Expand Down
2 changes: 2 additions & 0 deletions canal/canal.go
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,8 @@ func (c *Canal) prepareSyncer() error {
Dialer: c.cfg.Dialer,
Localhost: c.cfg.Localhost,
EventCacheCount: c.cfg.EventCacheCount,
FillZeroLogPos: c.cfg.FillZeroLogPos,

RowsEventDecodeFunc: func(event *replication.RowsEvent, data []byte) error {
pos, err := event.DecodeHeader(data)
if err != nil {
Expand Down
6 changes: 6 additions & 0 deletions canal/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,12 @@ type Config struct {
// the default value is 10240.
// if you table contain large columns, you can decrease this value to avoid OOM.
EventCacheCount int

// FillZeroLogPos enables dynamic LogPos calculation for MariaDB.
// When enabled, automatically adds BINLOG_SEND_ANNOTATE_ROWS_EVENT flag
// to ensure correct position calculation in MariaDB 11.4+.
// Only works with MariaDB flavor.
FillZeroLogPos bool
}

func NewConfigWithFile(name string) (*Config, error) {
Expand Down
12 changes: 6 additions & 6 deletions replication/binlogsyncer.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,11 +128,11 @@ type BinlogSyncerConfig struct {

EventCacheCount int

// MariaDBDynamicLogPos enables dynamic LogPos calculation for MariaDB.
// FillZeroLogPos enables dynamic LogPos calculation for MariaDB.
// When enabled, automatically adds BINLOG_SEND_ANNOTATE_ROWS_EVENT flag
// to ensure correct position calculation in MariaDB 11.4+.
// Only works with MariaDB flavor.
MariaDBDynamicLogPos bool
FillZeroLogPos bool

// SynchronousEventHandler is used for synchronous event handling.
// This should not be used together with StartBackupWithHandler.
Expand Down Expand Up @@ -516,8 +516,8 @@ func (b *BinlogSyncer) writeBinlogDumpCommand(p mysql.Position) error {
pos += 4

dumpCommandFlag := b.cfg.DumpCommandFlag
if b.cfg.MariaDBDynamicLogPos && b.cfg.Flavor == mysql.MariaDBFlavor {
// Add BINLOG_SEND_ANNOTATE_ROWS_EVENT flag when MariaDBDynamicLogPos is enabled.
if b.cfg.FillZeroLogPos && b.cfg.Flavor == mysql.MariaDBFlavor {
// Add BINLOG_SEND_ANNOTATE_ROWS_EVENT flag when FillZeroLogPos is enabled.
// This ensures the server sends ANNOTATE_ROWS_EVENT events which are needed
// for correct LogPos calculation in MariaDB 11.4+, where some events have LogPos=0.
dumpCommandFlag |= BINLOG_SEND_ANNOTATE_ROWS_EVENT
Expand Down Expand Up @@ -966,12 +966,12 @@ func (b *BinlogSyncer) handleEventAndACK(s *BinlogStreamer, e *BinlogEvent, need

// shouldCalculateDynamicLogPos determines if we should calculate LogPos dynamically for MariaDB events.
// This is needed for MariaDB 11.4+ when:
// 1. MariaDBDynamicLogPos is enabled
// 1. FillZeroLogPos is enabled
// 2. We're using MariaDB flavor
// 3. The event has LogPos=0 (indicating server didn't set it)
// 4. The event is not artificial (not marked with LOG_EVENT_ARTIFICIAL_F flag)
func (b *BinlogSyncer) shouldCalculateDynamicLogPos(e *BinlogEvent) bool {
return b.cfg.MariaDBDynamicLogPos &&
return b.cfg.FillZeroLogPos &&
b.cfg.Flavor == mysql.MariaDBFlavor &&
e.Header.LogPos == 0 &&
(e.Header.Flags&LOG_EVENT_ARTIFICIAL_F) == 0
Expand Down
Loading