Skip to content

Commit cddbb80

Browse files
authored
add explicit anchors
1 parent 497679b commit cddbb80

File tree

1 file changed

+7
-7
lines changed

1 file changed

+7
-7
lines changed

docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,27 +12,27 @@ import cdc_syncs from '@site/static/images/integrations/data-ingestion/clickpipe
1212

1313
This document describes how to control the sync of a database ClickPipe (Postgres, MySQL etc.) when the ClickPipe is in **CDC (Running) mode**.
1414

15-
## Overview
15+
## Overview {#overview}
1616

1717
Database ClickPipes have an architecture that consists of two parallel processes - pulling from the source database and pushing to the target database. The pulling process is controlled by a sync configuration that defines how often the data should be pulled and how much data should be pulled at a time. By "at a time", we mean one batch - since the ClickPipe pulls and pushes data in batches.
1818

1919
There are two main ways to control the sync of a database ClickPipe. The ClickPipe will start pushing when one of the below settings kicks in.
2020

21-
### Sync interval
21+
### Sync interval {#sync-interval}
2222
The sync interval of the pipe is the amount of time (in seconds) for which the ClickPipe will pull records from the source database. The time to push what we have to ClickHouse is not included in this interval.
2323

2424
The default is **1 minute**.
2525
Sync interval can be set to any positive integer value, but it is recommended to keep it above 10 seconds.
2626

27-
### Pull batch size
27+
### Pull batch size {#pull-batch-size}
2828
The pull batch size is the number of records that the ClickPipe will pull from the source database in one batch. Records mean inserts, updates and deletes done on the tables that are part of the pipe.
2929

3030
The default is **100,000** records.
3131

32-
### An exception: Long-running transactions on source
32+
### An exception: Long-running transactions on source {#exception-long-running-transactions-on-source}
3333
When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size.
3434

35-
### Configuring sync settings
35+
### Configuring sync settings {#configuring-sync-settings}
3636
You can set the sync interval and pull batch size when you create a ClickPipe or edit an existing one.
3737
When creating a ClickPipe it will be seen in the second step of the creation wizard, as shown below:
3838
<img src={create_sync_settings} alt="Create sync settings" />
@@ -43,12 +43,12 @@ When editing an existing ClickPipe, you can head over to the **Settings** tab of
4343
This will open a flyout with the sync settings, where you can change the sync interval and pull batch size:
4444
<img src={edit_sync_settings} alt="Edit sync settings" />
4545

46-
### Tweaking the sync settings to help with replication slot growth
46+
### Tweaking the sync settings to help with replication slot growth {#tweaking-the-sync-settings-replication-slot-growth}
4747
Let's talk about how to use these settings to handle a large replication slot of a CDC pipe.
4848
The pushing time to ClickHouse does not scale linearly with the pulling time from the source database. This can be leveraged to reduce the size of a large replication slot.
4949
By increasing both the sync interval and pull batch size, the ClickPipe will pull a whole lot of data from the source database in one go, and then push it to ClickHouse.
5050

51-
### Monitoring sync control behaviour
51+
### Monitoring sync control behavior {#monitoring-sync-control-behavior}
5252
You can see how long each batch takes in the **CDC Syncs** table in the **Metrics** tab of the ClickPipe. Note that the duration here includes push time and also if there are no rows incoming, the ClickPipe waits and the wait time is also included in the duration.
5353

5454
<img src={cdc_syncs} alt="CDC Syncs table" />

0 commit comments

Comments
 (0)