Skip to content

Commit fccb102

Browse files
add section on monitoring sync control
1 parent 4da8051 commit fccb102

File tree

3 files changed

+12
-0
lines changed

3 files changed

+12
-0
lines changed

docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ sidebar_label: 'Controlling syncs'
88
import edit_sync_button from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/edit_sync_button.png'
99
import create_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/create_sync_settings.png'
1010
import edit_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/sync_settings_edit.png'
11+
import cdc_syncs from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/cdc_syncs.png'
1112

1213
This document describes how to control the sync of a database ClickPipe (Postgres, MySQL etc.) when the ClickPipe is in **CDC (Running) mode**.
1314

@@ -41,3 +42,8 @@ When editing an existing ClickPipe, you can head over to the **Settings** tab of
4142

4243
This will open a flyout with the sync settings, where you can change the sync interval and pull batch size:
4344
<img src={edit_sync_settings} alt="Edit sync settings" />
45+
46+
### Monitoring sync control behaviour
47+
You can see how long each batch takes in the **CDC Syncs** table in the **Metrics** tab of the ClickPipe. Note that the duration here includes push time and also if there are no rows incoming, the ClickPipe waits and the wait time is also included in the duration.
48+
49+
<img src={cdc_syncs} alt="CDC Syncs table" />

docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ sidebar_label: 'Controlling syncs'
88
import edit_sync_button from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/edit_sync_button.png'
99
import create_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/create_sync_settings.png'
1010
import edit_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/sync_settings_edit.png'
11+
import cdc_syncs from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/cdc_syncs.png'
1112

1213
This document describes how to control the sync of a database ClickPipe (Postgres, MySQL etc.) when the ClickPipe is in **CDC (Running) mode**.
1314

@@ -46,3 +47,8 @@ This will open a flyout with the sync settings, where you can change the sync in
4647
Let's talk about how to use these settings to handle a large replication slot of a CDC pipe.
4748
The pushing time to ClickHouse does not scale linearly with the pulling time from the source database. This can be leveraged to reduce the size of a large replication slot.
4849
By increasing both the sync interval and pull batch size, the ClickPipe will pull a whole lot of data from the source database in one go, and then push it to ClickHouse.
50+
51+
### Monitoring sync control behaviour
52+
You can see how long each batch takes in the **CDC Syncs** table in the **Metrics** tab of the ClickPipe. Note that the duration here includes push time and also if there are no rows incoming, the ClickPipe waits and the wait time is also included in the duration.
53+
54+
<img src={cdc_syncs} alt="CDC Syncs table" />
36.2 KB
Loading

0 commit comments

Comments
 (0)