Skip to content

K8SPXC-1648: don't round up PVC size#2442

Open
pooknull wants to merge 5 commits intomainfrom
K8SPXC-1648
Open

K8SPXC-1648: don't round up PVC size#2442
pooknull wants to merge 5 commits intomainfrom
K8SPXC-1648

Conversation

@pooknull
Copy link
Copy Markdown
Contributor

@pooknull pooknull commented Apr 20, 2026

https://perconadev.atlassian.net/browse/K8SPXC-1648

DESCRIPTION

Problem:
When volumeSpec.persistentVolumeClaim.resources.requests.storage is set to 1.2Gi operator creates a statefulset with the same size, but 2Gi PVC is created. After updating the cr.yaml to the 2Gi the operator should recreate the statefulset with the new size, but it tries to update statefulset instead and gets the following error: ...update error: StatefulSet.apps "minimal-cluster-pxc" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ...

Cause:
The storage provisioners may create a 2Gi PVC for a 1.2Gi request because it allocates storage in larger chunks.
The operator logic rounded the requested size to GiB and then checked for an exact match, but the actual PVC may have a larger size than the value from the cr.yaml.
Because of that, after changing the CR to 2Gi, the operator wasn't able to check if PVC resize update is needed and tried to update the StatefulSet template instead.

Solution:
The operator now uses the exact value from cr.yaml and treats the pvc as correct when its actual size is greater than or equal to the requested size.
It also checks whether the StatefulSet template still has an old storage value. If the PVC already has enough space but the template is different, the operator deletes and recreates the StatefulSet instead of trying to update it

This PR should also fix https://perconadev.atlassian.net/browse/K8SPXC-1438

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?
  • Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Are all needed new/changed options added to the Helm Chart?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PXC version?
  • Does the change support oldest and newest supported Kubernetes version?

@pull-request-size pull-request-size Bot added the size/L 100-499 lines label Apr 20, 2026
@pooknull pooknull marked this pull request as ready for review April 27, 2026 13:42
Comment on lines +222 to +234
if configured.Cmp(requested) <= 0 {
if configured.Cmp(requested) == 0 {
return nil
}

log.Info("Deleting statefulset to reconcile volume claim template", "configured", configured.String(), "requested", requested.String(), "actual", actual.String())
if err := r.client.Delete(ctx, sts, client.PropagationPolicy("Orphan")); err != nil {
if !k8serrors.IsNotFound(err) {
return errors.Wrapf(err, "delete statefulset/%s", sts.Name)
}
}
return nil
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't understand why this is needed, wasn't the issue caused by rounding the value up and fixed by removing rounding logic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this part the operator tries to update the StatefulSet template with the value specified in the cr.yaml. If we are going to remove this part then we should fix the problem with sts update by setting the actual size of the PVC in the pxc.StatefulSet function. So I think that the current version is a cleaner fix

@egegunes egegunes added this to the v1.20.0 milestone Apr 28, 2026
@pooknull pooknull requested a review from egegunes April 28, 2026 10:35
@JNKPercona
Copy link
Copy Markdown
Collaborator

Test Name Result Time
auto-tuning-8-0 failure 00:00:00
allocator-8-0 failure 00:00:00
allocator-8-4 failure 00:00:00
backup-storage-tls-8-0 failure 00:00:00
cross-site-8-0 failure 00:00:00
cross-site-proxysql-8-0 failure 00:00:00
cross-site-proxysql-8-4 failure 00:00:00
custom-users-8-0 failure 00:00:00
demand-backup-cloud-8-0 failure 00:00:00
demand-backup-cloud-8-4 failure 00:00:00
demand-backup-cloud-pxb-8-0 failure 00:00:00
demand-backup-encrypted-with-tls-5-7 failure 00:00:00
demand-backup-encrypted-with-tls-8-0 skipped 00:00:00
demand-backup-encrypted-with-tls-8-4 skipped 00:00:00
demand-backup-encrypted-with-tls-pxb-5-7 skipped 00:00:00
demand-backup-encrypted-with-tls-pxb-8-0 skipped 00:00:00
demand-backup-encrypted-with-tls-pxb-8-4 skipped 00:00:00
demand-backup-8-0 skipped 00:00:00
demand-backup-flow-control-8-0 skipped 00:00:00
demand-backup-flow-control-8-4 skipped 00:00:00
demand-backup-parallel-8-0 skipped 00:00:00
demand-backup-parallel-8-4 skipped 00:00:00
demand-backup-without-passwords-8-0 skipped 00:00:00
demand-backup-without-passwords-8-4 skipped 00:00:00
extra-pvc-8-0 skipped 00:00:00
haproxy-5-7 skipped 00:00:00
haproxy-8-0 skipped 00:00:00
haproxy-8-4 skipped 00:00:00
init-deploy-5-7 skipped 00:00:00
init-deploy-8-0 skipped 00:00:00
limits-8-0 skipped 00:00:00
monitoring-2-0-8-0 skipped 00:00:00
monitoring-pmm3-8-0 skipped 00:00:00
monitoring-pmm3-8-4 skipped 00:00:00
one-pod-5-7 skipped 00:00:00
one-pod-8-0 skipped 00:00:00
pitr-8-0 skipped 00:00:00
pitr-8-4 skipped 00:00:00
pitr-pxb-8-0 skipped 00:00:00
pitr-pxb-8-4 skipped 00:00:00
pitr-gap-errors-8-0 skipped 00:00:00
pitr-gap-errors-8-4 skipped 00:00:00
proxy-protocol-8-0 skipped 00:00:00
proxy-switch-8-0 skipped 00:00:00
proxysql-sidecar-res-limits-8-0 skipped 00:00:00
proxysql-scheduler-8-0 skipped 00:00:00
pvc-resize-5-7 skipped 00:00:00
pvc-resize-8-0 skipped 00:00:00
recreate-8-0 skipped 00:00:00
restore-to-encrypted-cluster-8-0 skipped 00:00:00
restore-to-encrypted-cluster-8-4 skipped 00:00:00
restore-to-encrypted-cluster-pxb-8-0 skipped 00:00:00
restore-to-encrypted-cluster-pxb-8-4 skipped 00:00:00
scaling-proxysql-8-0 skipped 00:00:00
scaling-8-0 skipped 00:00:00
scheduled-backup-5-7 skipped 00:00:00
scheduled-backup-8-0 skipped 00:00:00
scheduled-backup-8-4 skipped 00:00:00
security-context-8-0 skipped 00:00:00
smart-update1-8-0 skipped 00:00:00
smart-update1-8-4 skipped 00:00:00
smart-update2-8-0 skipped 00:00:00
smart-update2-8-4 skipped 00:00:00
smart-update3-8-0 skipped 00:00:00
sst-retry-limit-8-0 skipped 00:00:00
sst-retry-limit-8-4 skipped 00:00:00
storage-8-0 skipped 00:00:00
tls-issue-cert-manager-ref-8-0 skipped 00:00:00
tls-issue-cert-manager-8-0 skipped 00:00:00
tls-issue-self-8-0 skipped 00:00:00
upgrade-consistency-8-0 skipped 00:00:00
upgrade-consistency-8-4 skipped 00:00:00
upgrade-haproxy-5-7 skipped 00:00:00
upgrade-haproxy-8-0 skipped 00:00:00
upgrade-proxysql-5-7 skipped 00:00:00
upgrade-proxysql-8-0 skipped 00:00:00
users-5-7 skipped 00:00:00
users-8-0 skipped 00:00:00
users-scheduler-8-4 skipped 00:00:00
validation-hook-8-0 skipped 00:00:00
Summary Value
Tests Run 12/80
Job Duration 00:16:07
Total Test Time N/A

commit: e171efa
image: perconalab/percona-xtradb-cluster-operator:PR-2442-e171efa4

configured: "1Gi",
actual: "6G",
},
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it make sense to add the case with configured=1.2Gi, actual=2Gi requested=2Gi ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L 100-499 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants