Best Practices for CloudStack Upgrade (Current Version: 4.19.3.0) – Request for Recommendations

[shakibshalim74]

Hello,

We are planning to upgrade our CloudStack environment and would appreciate guidance on best practices and rollback strategies.
Current Setup:
CloudStack: 4.19.3.0
Number of Management Servers: 3
Number of Database Server: 1
KVM Hosts: 30
Number of running instances : 400

Looking for recommendations on:
Best-practice upgrade sequence (Management, Data Base, KVM hosts)
Known issues when upgrading from 4.19.3.0
Key validation checks post-upgrade
Reliable rollback methods (DB backup/restore, KVM agent rollback, System VM/VR template rollback, config and certificate backups)

Some question regarding the upgrade process:

We plan to upgrade the Management and DB servers first according to the official upgrade documentation.
If we start upgraded management service first, and then upgrade KVM hosts one by one, are there any caveats? We are asking this question as the official upgrade documentation suggests starting the management server at the very end of the upgradation process after all agents are being upgraded.
Is it safe to run upgraded Management+DB services with hypervisors still on the older version? or should we need all Hypervisor Agents to be upgraded before starting Management services?
Any best-practice recommendations or real-world feedback would be highly appreciated.

[kiranchavala]

@shakibshalim74 what's the management server os and kvm host os details

[shakibshalim74]

My Management server OS is Ubuntu 22.04.5, We have some KVM host on ubuntu 20.04, ubuntu 22.04 and 24.04 as well.

[rajujith]

@shakibshalim74
Try this in staging before attempting in production.

Refer https://docs.cloudstack.apache.org/en/4.20.2.0/upgrading/upgrade/upgrade-4.19.html

Full backup of cloud and cloud_usage databases, including routines.

$ mysqldump -u root -p -R cloud > cloud-backup_$(date +%Y-%m-%d-%H%M%S)
$ mysqldump -u root -p cloud_usage > cloud_usage-backup_$(date +%Y-%m-%d-%H%M%S)

Snapshot / backup all Management Server VMs, Preserve:

/etc/cloudstack/management
SSL / LDAPS keystores

On all 3 Management Servers:

Upgrade CloudStack management packages to 4.20.2
Do NOT start the service yet

This ensures all MS nodes run the same binaries before schema changes begin.

Start ONE Management Server
This server:

Detects older schema version
Performs automatic schema upgrades on : cloud,cloud_usage

Monitor:

management-server.log

Verify UI/API access

Start Remaining Management Servers

Start the Management service on the remaining MS nodes
Validate:

All MS are Up

KVM Host Upgrade

Upgrade cloudstack-agent on one host, then upgrade all other hosts with automation or other tools available.
Restart agent only
Running VMs are not impacted

System VM Template Upgrade

Reboot / redeploy:

VR
CPVM
SSVM

RollBack:
Disable the zone/s.
Delete any SystemVM,VRs created after the upgrade
Stop all Management Servers
Restore cloud + cloud_usage DB backups
Downgrade management packages
Downgrade cloudstack-agent on all KVM hosts
Start Management Servers

Enable the zone, redeploy:

VR
CPVM
SSVM

[kiranchavala]

@shakibshalim74

you can also refer to the past discussions. I would advise to test the upgrade in a staging environment before applying it in production envrionment

#12054
#11044