PG16-K8s cannot be deployed to Canonical K8s. Error: calling self.fix_leader_annotation() (PG14-K8s works)

[taurus-forever]

Steps to reproduce

1. Launch LXD with enough disk/CPU/RAM: lxc launch --vm ubuntu:24.04 -c limits.cpu=12 -c limits.memory=32GiB -d root,size=200GiB myc8s
2. Deploy Canonical K8s inside newly started VM using https://canonical-charmed-postgresql-k8s.readthedocs-hosted.com/16/how-to/deploy/canonical-k8s/
3. try to deploy PG16: juju deploy postgresql-k8s --channel 16/edge --trust (r613 is failing). PG14 woks well (r495)!

Expected behavior

PG16 deployed without errors

Actual behavior

Charm stuck in awaiting for cluster to start:

root@2xenrico:~# juju status
Model   Controller  Cloud/Region  Version  SLA          Timestamp
test16  ck8s        ck8s          3.6.7    unsupported  16:35:03Z

App             Version  Status   Scale  Charm           Channel  Rev  Address         Exposed  Message
postgresql-k8s  16.9     waiting      1  postgresql-k8s  16/edge  613  10.152.183.219  no       awaiting for cluster to start

Unit               Workload  Agent      Address     Ports  Message
postgresql-k8s/0*  waiting   executing  10.1.0.172         awaiting for cluster to start

Versions

Operating system: 24.04
Juju CLI: 3.6.7
Juju agent: 3.6.7
Charm revision: postgresql-k8s 16/edge r613
Canonical K8s: k8s v1.32.5 3581 1.32-classic/stable canonical✓ classic

Log output

unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log ops 2.22.0 up and running.
unit-postgresql-k8s-0: 16:42:03 WARNING unit.postgresql-k8s/0.juju-log /var/lib/juju/agents/unit-postgresql-k8s-0/charm/src/charm.py:220: DeprecationWarning: JujuVersion.from_environ() is deprecated, use self.model.juju_version instead
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log no relation on 'tracing': tracing not ready
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Starting root trace with id='b46b424fd89d9f9b71781b7ed3d9761a'.
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event <PebbleReadyEvent on <ops.model.Container object at 0x78a44f5de480> via PostgresqlOperatorCharm/on/postgresql_pebble_ready[26]>.
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Building PostgreSQL parameters for profile='production' and available_memory=33541885952
unit-postgresql-k8s-0: 16:42:03 INFO unit.postgresql-k8s/0.juju-log Updating Patroni config file
unit-postgresql-k8s-0: 16:42:04 DEBUG unit.postgresql-k8s/0.juju-log Early exit update_config: Workload not started yet
unit-postgresql-k8s-0: 16:42:04 INFO unit.postgresql-k8s/0.juju-log Updated health checks
unit-postgresql-k8s-0: 16:42:04 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:05 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:06 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:07 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:08 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:09 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:10 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:11 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:12 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:13 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring on_postgresql_pebble_ready: Waiting for cluster to start
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring <PebbleReadyEvent on <ops.model.Container object at 0x78a44f5de480> via PostgresqlOperatorCharm/on/postgresql_pebble_ready[26]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event <ConfigChangedEvent via PostgresqlOperatorCharm/on/config_changed[56]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Defer on_config_changed: cluster not initialised yet
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring <ConfigChangedEvent via PostgresqlOperatorCharm/on/config_changed[56]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event <RelationChangedEvent app='postgresql-k8s' unit=None on <ops.model.Relation database-peers:2> via PostgresqlOperatorCharm/on/database_peers_relation_changed[71]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:15 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:16 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:17 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:18 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:19 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:20 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:21 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:22 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:23 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Deferring on_postgresql_pebble_ready: Waiting for primary endpoint to be ready
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Deferring <RelationChangedEvent app='postgresql-k8s' unit=None on <ops.model.Relation database-peers:2> via PostgresqlOperatorCharm/on/database_peers_relation_changed[71]>.
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log _initialized_cluster failed on _peer_relation_changed
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Emitting Juju event update_status.
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log pgdata free disk space: 1003884544 out of 1020702720, ratio of 0.9835229438792913
unit-postgresql-k8s-0: 16:42:24 ERROR unit.postgresql-k8s/0.juju-log calling self.fix_leader_annotation()
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log on_update_status early exit: Unit is in Blocked/Waiting status
unit-postgresql-k8s-0: 16:42:24 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-7582.

This message was autogenerated

[taurus-forever]

For the history: I have tried to deploy Canonical K8s one more time using newer snap:

k8s   v1.32.6  3789  1.32-classic/stable  canonical✓  classic

I do not see error calling self.fix_leader_annotation() anylonger, but PG16 deployment still stuck and PG14 works well.
P.S. No errors in logs.

[taurus-forever]

This is the same issue as canonical/postgresql-operator#1336 on PG VM.
TL;DR: PostgreSQL (pg_init) doesn't not bootstrap cluster due to lost+found ext4 folder in K8s mount point. It forces to provide an empty folder.

Refactoring WIP: https://docs.google.com/document/d/1eCVf0OH0BZ6GAt9bhxhgObWubPWttn5IZa0GXVtdvJo/edit?tab=t.0