Contrary to the statement in the documentation that the data is deleted from the destination, copying a chunk multiple times results in duplicates in the destination.
Example:
A time range was staged and copied for the first time:
./timescaledb-backfill stage --source "pq-connect-string" --target "pq-connect-string" --from "2026-07-12" --until "2026-07-13"
timescaledb-backfill v0.13.0
Staged 1 chunks to copy.
Execute the copy command to migrate the data.
./timescaledb-backfill copy --source "pq-connect-string" --target "pq-connect-string"
timescaledb-backfill v0.13.0
Copying 1 chunks with 8 workers
[1/1] Copied chunk "_timescaledb_internal"."_hyper_1_3192_chunk" in 1.67s (56.9MB/s)
Copied 94.8MB from 1 chunks in 1.98s.
Execute the verify command to assert data integrity.
./timescaledb-backfill verify --source "pg-connect-string" --target "pq-connect-string"
timescaledb-backfill v0.13.0
Verifying 1 chunks with 8 workers
[1/1] Verified chunk "_timescaledb_internal"."_hyper_1_3192_chunk" in 3.09s
Verifed 1 chunks in 3.21s.
Execute the clean command to remove the backfill administrative schema from the target database.
./timescaledb-backfill clean --target "pq-connect-string"
timescaledb-backfill v0.13.0
Removed backfill administrative schema from target database
Then the same time range is staged and copied again and the verify process then complains about a difference and display exactly twice the number of expected entries:
./timescaledb-backfill verify --source "pq-connect-string" --target "pq-connect-string"
timescaledb-backfill v0.13.0
Verifying 1 chunks with 8 workers
[1/1] Chunk verification failed, source="_timescaledb_internal"."_hyper_1_3192_chunk" target="_timescaledb_internal"."_hyper_1_699_chunk" diff
diff
--- original
+++ modified
@@ -8,7 +8,7 @@
value: '958790'
sum: {}
count:
- pv: 34673080
- ts: 34673080
- value: 34673080
-total_count: 34673080
+ pv: 69346160
+ ts: 69346160
+ value: 69346160
+total_count: 69346160
Verifed 1 chunks in 7.14s.
Execute the 'clean' command to remove the backfill administrative schema from the target database.
An excerpt from the chunk in the target database where a duplicate is located looks like:
myhypertable
│
├── _hyper_1_699_chunk
│ (2026-07-12 12:00 - 24:00)
│
│ UNIQUE(pv,ts)
│
│ logically visible
│
│
└── compress_hyper_2_699_chunk
│
├── pv=A
│
├── seq=10
│ min=23:58:19
│ max=23:59:59
│
├── seq=20
│ min=23:56:39
│ max=23:58:19
Could the reason for this be that the data is inserted at the “lowest level”? In this process, data is written not to the logical chunk (_timescaledb_internal.hyper<...>) but to the compressed chunk structure.
This bypasses the Hypertable index and is therefore not subject to any constraints.
We don't think this is the expected behavior. Or is it?
The statements in the documentation are somewhat confusing:
- append only
- delete existing data
One could think these are contrary statements.
Best regards
Oliver
Contrary to the statement in the documentation that the data is deleted from the destination, copying a chunk multiple times results in duplicates in the destination.
Example:
A time range was staged and copied for the first time:
Then the same time range is staged and copied again and the verify process then complains about a difference and display exactly twice the number of expected entries:
An excerpt from the chunk in the target database where a duplicate is located looks like:
Could the reason for this be that the data is inserted at the “lowest level”? In this process, data is written not to the logical chunk (_timescaledb_internal.hyper<...>) but to the compressed chunk structure.
This bypasses the Hypertable index and is therefore not subject to any constraints.
We don't think this is the expected behavior. Or is it?
The statements in the documentation are somewhat confusing:
One could think these are contrary statements.
Best regards
Oliver