PXB-3747: report backup size and uncompressed size

[satya-bodapati]

Summary

xtrabackup now embeds two byte-accurate size fields in xtrabackup_info
and mirrors them to the error log:

• backup_size (bytes) — final bytes the backup wrote to disk
  (or to the stream). Always reported.
• uncompressed_backup_size (bytes) — logical pre-compression
  bytes (data + redo + metadata + server-encrypted InnoDB files +
  RocksDB SSTs that skip the compressor). Reported only when
  --compress is used.

Both values are byte-perfect against the actual on-disk artifact for
every combination of --compress{,-lz4,-zstd} / --encrypt /
--stream=xbstream / --extra-lsndir / incremental / sparse.
Strict-equality asserts in t/backup_size_basic.sh and
t/backup_size_compress.sh cover 15+ scenarios end-to-end.

JIRA: https://perconadev.atlassian.net/browse/PXB-3747

How to use this feature

No new flags or options — the behavior is enabled for every backup.

1. Final backup size (backup_size) — always present in
   xtrabackup_info, e.g.:

   backup_size = 129456789
   

   and logged:

   Backup size: 123.45 MiB (129456789 bytes)
   

2. Uncompressed size under --compress (uncompressed_backup_size) —
   added only when --compress{,-lz4,-zstd} is used:

   uncompressed_backup_size = 479123456
   

   along with two extra log lines:

   Uncompressed backup size: 456.78 MiB (479123456 bytes)
   Compression ratio: 3.70x
   

   This helps operators estimate the disk space required to extract
   a compressed backup before running xbstream -x /
   xtrabackup --decompress, and makes backups taken at different
   dates comparable regardless of compressibility.

3. Where xtrabackup_info lives — the copy inside the backup
   directory follows the transport the user picked: it is compressed
   under --compress, encrypted under --encrypt, and wrapped in
   the xbstream under --stream=xbstream. To always obtain an
   unencrypted, uncompressed copy of xtrabackup_info, pass
   --extra-lsndir=<dir> — the file written there is plaintext
   regardless of --compress / --encrypt.

4. Error-log output — the two (or three, under --compress)
   lines above appear at the end of the backup, just before the
   "completed OK!" banner.

Commits

Split into two focused, bisect-friendly commits:

1. PXB-3747 : Report final on-disk backup_size in xtrabackup_info
   — adds backup_size with zero --compress awareness; includes
   the shared strict-size test infrastructure.
2. PXB-3747 : Report uncompressed_backup_size for --compress backups
   — adds uncompressed_backup_size, the "Uncompressed backup size"
   log line, and the "Compression ratio" log line, wired up only
   for --compress runs.

Commit 1 design — backup_size

The datasink pipeline has exactly one authoritative component: the
leaf. Every byte that lands in the backup artifact passes
through ds_local / ds_stdout / ds_fifo. Wrapper datasinks
(encrypt, xbstream, buffer, tmpfile, compress) only transform and
forward bytes.

    ds_data (head)
         |  ds_write()
         v
    [ wrappers: xbstream / encrypt / compress / ... ]
         |  pipe_ctxt
         v
    leaf (ds_local | ds_stdout | ds_fifo)   authoritative byte count
         |
         v
    my_write() / fwrite()

Rather than adding one-off get_foo() vtable slots per metric, the
patch introduces a tiny per-datasink metrics API: each datasink
can publish named {name, value} pairs via an optional
report_metrics slot, and callers read one off a specific node
with ds_find_metric. The three leaves publish an atomic
bytes_written counter bumped on every successful write.
get_final_backup_size() walks to the leaf and reads it;
backup_size = N is written to xtrabackup_info and logged.

Tablespace_map::serialize() and the transition-key dumper are
moved ahead of backup_finish() so their bytes are included in
the reported backup_size.

Commit 2 design — uncompressed_backup_size

Why two data pipelines exist

xtrabackup maintains two top-level data pipelines because not
every file benefits from compression:

• ds_data — normal InnoDB tablespaces, redo, metadata.
  Compressed when --compress is on.
• ds_uncompressed_data — server-encrypted InnoDB tablespaces
  (re-compressing ciphertext is wasteful) and RocksDB SST files
  (already compressed internally). Bypasses the compress wrapper.

Both pipelines share their encrypt / xbstream / leaf tail as an
init-time context-sharing optimization:

    ds_data              --> buffer --> compress --> encrypt --> leaf
                                                                  ^
                                                                  |
    ds_uncompressed_data -----------------------> encrypt --------+

Why the counter cannot live on any ds_ctxt_t

The shared tail makes "attach a counter to the ctxt" broken no
matter which ctxt is picked:

• Leaf ctxt. Under --stream=xbstream, ds_data, ds_redo,
  ds_meta, and ds_uncompressed_data all terminate at the same
  leaf. A leaf-ctxt counter would sum bytes from every pipeline,
  and for the compressed pipeline those bytes are post-compression
  — neither the logical total nor the on-disk total.
• Encrypt / xbstream / buffer ctxts. ds_data and
  ds_uncompressed_data share these too. Counting there mixes
  post-compression bytes (from ds_data) with raw bytes (from
  ds_uncompressed_data); the sum is meaningless.

Per-file ds_file_t handle is the right anchor

The counter attaches where a single logical write unambiguously
enters exactly one pipeline — on the per-file handle returned by
the top-level ds_open(). Each top-level open creates a fresh
ds_file_t, the caller decides whether that file's bytes count,
and ds_write / ds_write_sparse bump the counter with the
length supplied at the entry, before any wrapper transforms it:

    xtrabackup top-level open
         |  ds_open_track_uncomp(..., xb_global_track_uncomp_bytes())
         |                             |
         |                             +-- nullptr when !--compress
         v
    ds_file_t { uncomp_bytes -> xb_uncomp_bytes_counter }
         |  ds_write(len) / ds_write_sparse(packed_len)
         |     if (file->uncomp_bytes) uncomp_bytes->add_uncompressed(len)
         v
    [ compress / encrypt / buffer / xbstream wrappers ]
         |
         v
    leaf  (already counted as backup_size by commit 1)

    xb_uncomp_bytes_counter.get_uncompressed_backup_size()
         |
         v
    uncompressed_backup_size  (xtrabackup_info + error log)

Tracking is opt-in and scoped to the top-level opens on ds_data
/ ds_redo / ds_meta / ds_uncompressed_data, so the hot-path
cost is one atomic add per write on --compress runs; non-compress
runs skip even that. The same counter aggregates both pipelines,
so server-encrypted IBDs and RocksDB SSTs are counted exactly
once, at their logical size.

xb_global_track_uncomp_bytes() returns a real counter only when
--compress is active, so non-compress runs emit neither the
field nor the log lines.

Test plan

• t/backup_size_basic.sh — --target-dir, --stream=xbstream,
  --encrypt on both, incremental chains, sparse tablespaces, full
  restore round-trip. Every scenario asserts backup_size matches
  the on-disk / on-stream byte count exactly; uncompressed_backup_size
  must not be present.
• t/backup_size_compress.sh — --compress target-dir,
  --compress xbstream, --compress + --encrypt, incremental
  chains with compression, RocksDB, server-encrypted InnoDB,
  redo-log encryption, sparse + --compress, and --extra-lsndir.
  Every scenario asserts backup_size matches the compressed
  on-disk size and, after decompression, uncompressed_backup_size
  matches the decompressed tree size byte-for-byte.
• suites/reducedlock/backup_size_reduced.sh — reduced-lock
  • concurrent TRUNCATE edge case (zero-size tablespace).
• -s compression — 12/12 runnable tests pass; shared
  strict-size helpers live in inc/common.sh.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[satya-bodapati]

https://pxb.cd.percona.com/view/PXB%208.1/job/percona-xtrabackup-8.1-multijob/144/

[Copilot]

Pull request overview

This PR adds byte-accurate backup size reporting to xtrabackup by plumbing size counters through the datasink pipeline and emitting the results in xtrabackup_info and the error log. It also introduces end-to-end shell tests that assert strict equality between reported sizes and on-disk/on-stream artifacts across compressed/uncompressed, encrypted, streamed, incremental, and sparse scenarios.

Changes:

• Add backup_size reporting based on a leaf datasink byte counter (local/stdout/fifo) and log it at backup completion.
• Add uncompressed_backup_size reporting for --compress via per-file metrics tracked at top-level ds_open() entrypoints (pre-compression logical bytes).
• Add shared test helpers and new test suites validating strict byte-perfect invariants across many backup configurations.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
storage/innobase/xtrabackup/test/t/backup_size_compress.sh	New end-to-end coverage validating backup_size + uncompressed_backup_size under --compress across multiple scenarios.
storage/innobase/xtrabackup/test/t/backup_size_basic.sh	New end-to-end coverage validating backup_size for non-compress backups and asserting absence of uncompressed_backup_size.
storage/innobase/xtrabackup/test/suites/reducedlock/backup_size_reduced.sh	New reduced-lock TRUNCATE regression test for byte-perfect backup_size.
storage/innobase/xtrabackup/test/inc/common.sh	Adds shared helpers for parsing xtrabackup_info and performing strict size assertions.
storage/innobase/xtrabackup/src/xtrabackup.h	Declares size-reporting APIs and introduces xb_get_metrics() helper for conditional tracking under --compress.
storage/innobase/xtrabackup/src/xtrabackup.cc	Implements size getters, ensures tracked opens for key backup artifacts, and reorders operations so size sampling includes late-written files.
storage/innobase/xtrabackup/src/space_map.cc	Switches tablespace map output to tracked open for inclusion in uncompressed metrics when compressing.
storage/innobase/xtrabackup/src/redo_log.cc	Switches redo logfile creation to tracked open for uncompressed metrics.
storage/innobase/xtrabackup/src/keyring_plugins.cc	Switches key dump output to tracked open for uncompressed metrics.
storage/innobase/xtrabackup/src/ds_xbstream.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_tmpfile.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_stdout.cc	Adds leaf byte counting + get_bytes_written implementation for stdout leaf and ensures file initialization.
storage/innobase/xtrabackup/src/ds_local.cc	Adds leaf byte counting + get_bytes_written implementation for local leaf (including sparse trailing-byte fix accounting).
storage/innobase/xtrabackup/src/ds_fifo.cc	Adds leaf byte counting + get_bytes_written implementation for FIFO leaf.
storage/innobase/xtrabackup/src/ds_encrypt.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decrypt.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress_zstd.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress_lz4.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress_zstd.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress_lz4.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_buffer.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/datasink.h	Extends datasink interface with get_bytes_written, introduces metrics plumbing on ds_file_t, and adds tracked open helpers.
storage/innobase/xtrabackup/src/datasink.cc	Implements tracked open + metrics accumulation, adds ds_leaf(), and adjusts ds_create initialization.
storage/innobase/xtrabackup/src/backup_mysql.cc	Embeds backup_size and conditionally appends uncompressed_backup_size into xtrabackup_info.
storage/innobase/xtrabackup/src/backup_copy.cc	Switches top-level opens to tracked opens and logs human-readable size summaries + compression ratio.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[satya-bodapati]

Pushed follow-up commit 7cc89789e5b — PXB-3747 : Addressing review comments. Will squash before merge.

What it addresses:

1. fs_support_punch_hole initialization — ds_ctxt_t::pipe_ctxt now defaults to nullptr in the struct, and the five my_malloc-based sinks (local, stdout, xbstream, buffer, tmpfile) use MY_FAE | MY_ZEROFILL. xbstream_init() advertises fs_support_punch_hole = true since its wire format round-trips sparse chunks natively. ds_create() no longer overrides the capability bit.
2. local_close() counts the 1-byte trailing-hole write into bytes_written, so sparse files under --target-dir no longer under-report by 1 byte per file.
3. "Finsh" → "Finish" in backup_copy.cc.
4. Reflowed the report_backup_size() doc comment so the "Compression ratio: inf" example reads as a single sentence.
5. sum_file_bytes() in inc/common.sh early-returns 0 when the argument is not a directory, silencing the stray find(1) stderr message.

Verified by rebuilding in Debug and running t/backup_size_basic.sh and t/backup_size_compress.sh — both pass.

[satya-bodapati]

Follow-up on c935a9d (force-pushed): dropped the redundant packed_len loop in ds_write_sparse(). The len parameter is already the packed (hole-excluded) payload size — callers pre-pack the buffer (dst_pos in write_ibd_buffer, chunk.length in xbstream.cc) and local_write_sparse walks the sparse_map writing exactly that many bytes in total. Now we just add_uncompressed(len). Verified with both t/backup_size_basic.sh and t/backup_size_compress.sh (byte-exact size assertions).

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jakub-nowakowski-percona]

Approved with optional remarks.

[satya-bodapati]

Follow-up commit 8d20d4b — PXB-3747 : Address Jakub's review comments (will squash before merge).

What it addresses from Jakub's review round:

• J2 (datasink.h pointer init) — every remaining pointer member in ds_ctxt_t (datasink, root, ptr) and all five in ds_file_t (ptr, path, datasink, ctxt, uncomp_bytes) is now default-initialised to nullptr in-class. Newly constructed handles no longer depend on the allocation site remembering MY_ZEROFILL / explicit field init.
• J3 + J5 (ds_metric::name and ds_find_metric doc) — ds_metric::name and ds_find_metric's name parameter are now std::string_view. The ownership comment and the "(strcmp comparison)" implementation-leak are gone. ds_find_metric uses m.name == name; strcmp / <cstring> removed from datasink.cc. Producers stay the same — string literals convert implicitly.
• J4 (report_metrics out param) — the vtable slot is now std::vector<ds_metric> &out. The three leaf implementations (ds_local, ds_stdout, ds_fifo) use out.push_back(...) accordingly; the single caller (ds_find_metric) passes a stack-local vector. Other vtables' nullptr /* report_metrics */ placeholders are unaffected.

Left as-is with the existing replies standing: J1 (ALWAYS_INLINE) and J6 (MY_ZEROFILL).

C1 (Compression ratio 3.70 vs 3.70x): the PR description already shows 3.70x; the Commit-2 message example will be fixed at squash time.

Verified: Debug rebuild clean; t/backup_size_basic.sh and t/backup_size_compress.sh both pass (byte-exact size assertions).