Backups can fail silently - Failed backup notifications are not being sent #140
Description
The latest version of the backup-container appears to be failing to send failed backup notifications.
We encountered an issue where the backup volume filled up and was causing backups to fail.
Backing up 'wallet-idim/agent_idim_wallet' to '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress' ...
gzip: stdout: No space left on device
removed '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress'
[!!ERROR!!] - Failed to backup postgres=wallet-idim/agent_idim_wallet.
Elapsed time: 0h:0m:56s - Status Code: 141
The notifications for these backup failures was never sent, however the subsequent verification of a backup that was corrupted by this issue did cause a failed verification notification to be sent.
This means backups can fail silently and if there is no subsequent failed verification notification the issue can go unnoticed.