K8SPXC-1800 | run prepare job for PITR#2449
Open
mayankshah1607 wants to merge 6 commits intomainfrom
Open
Conversation
Signed-off-by: Mayank Shah <mayank.shah@percona.com>
Signed-off-by: Mayank Shah <mayank.shah@percona.com>
Signed-off-by: Mayank Shah <mayank.shah@percona.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| echo ${gtid} | ||
| } | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
Suggested change
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the PXC restore controller so PITR restores run the “prepare cluster” job (needed for cross-cluster replication and password-changed backups), and extends PITR E2E coverage to exercise restore-from-source onto a freshly recreated cluster with changed system user passwords.
Changes:
- Move PITR execution to occur after the “Preparing Cluster” stage by running it from the
RestorePrepareClusterstate handler. - Extend PITR E2E scripts to support dynamic root passwords and add a new case restoring onto a recreated cluster.
- Update PITR test cluster manifest finalizers to ensure PVC cleanup during cluster deletion/recreate flows.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pkg/controller/pxcrestore/controller.go | Adjust restore state machine so PITR is triggered after the prepare job completes. |
| e2e-tests/pitr/run | Add password randomization helper and a new PITR restore-on-new-cluster scenario. |
| e2e-tests/pitr/conf/pitr.yml | Add PVC-deletion finalizers to support delete/recreate in PITR E2E. |
| e2e-tests/functions | Pass root password into PITR recovery checks and wait for new intermediate restore states. |
Comments suppressed due to low confidence (1)
pkg/controller/pxcrestore/controller.go:355
- PITR restore for clusters with version < 1.18.0 appears to be broken: PITR job creation/state transition was moved into reconcileStatePrepareCluster, but reconcileStateRestore routes <1.18.0 restores directly to RestoreStartCluster (skipping RestorePrepareCluster), so RestorePITR is never reached and the PITR job won't run. Consider keeping the PITR branching in reconcileStateRestore for <1.18.0 (or otherwise ensuring PITR flows still create/run the PITR job when PrepareCluster is skipped).
if cluster.CompareVersionWith("1.18.0") >= 0 {
log.Info("preparing cluster", "cluster", cr.Spec.PXCCluster)
cr.Status.State = api.RestorePrepareCluster
} else {
log.Info("starting cluster", "cluster", cr.Spec.PXCCluster)
cr.Status.State = api.RestoreStartCluster
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2282
to
2286
| wait_backup_restore ${restore} "Restoring" | ||
|
|
||
| wait_backup_restore ${restore} "Preparing Cluster" | ||
|
|
||
| wait_backup_restore ${restore} "Point-in-time recovering" |
Comment on lines
+281
to
+283
| desc "[CASE 4] delete PXC cluster ${cluster} and wait for PVCs" | ||
| kubectl_bin delete pxc ${cluster} | ||
| desc "[CASE 4] wait for PVCs (app.kubernetes.io/instance=${cluster}) to be deleted" |
Member
Author
There was a problem hiding this comment.
kubectl delete blocks until the object is gone
Signed-off-by: Mayank Shah <mayank.shah@percona.com>
mayankshah1607
requested review from
egegunes,
eleo007,
gkech,
hors,
jvpasinatto,
nmarukovich,
oksana-grishchenko,
pooknull and
valmiranogueira
as code owners
Collaborator
commit: b47912f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGE DESCRIPTION
Problem:
Prepare job does not run when PITR is specified. As a result, PXC pods do not come up for cross-cluster replication or if the passwords are changed since the backup.
Cause:
Prepare job is skipped in PITR step.
Solution:
Run prepare job for PITR
CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability