Need help with VM console

[SviridoffA]

Today I noticed that on 4 of my nodes, noVNC connections are not working (possibly preceded by a network issue, but I'm not sure about that). When connecting, a browser tab opens, after which there's just a connection status bar, but the actual connection to the console doesn't occur. Neither restarting the vmconsole VM nor restarting the CloudStack agent on the hypervisor helps. Only either restarting the virtual machine or simply migrating it helps, after which the connection works normally. In this way, I fixed problem on all current clients VMs, but left my own virtual machines in this state for further investigation. Please advise what I can check? What logs and what key phrases to look for? Or perhaps someone has already encountered something similar and can help figure out this issue.

[DaanHoogland]

@SviridoffA , did this start happening on an upgrade of

• ACS
• a network element
• software items on hosts?

(sorry to not have a ready baked answer ;)

[SviridoffA]

@DaanHoogland
No, I haven't performed any updates in the last couple of months. After the last update, the nodes were rebooted, after which I checked the connections since I use my own HA automation tied to specially prepared and host-bound VMs. After the update, the vmconsole worked fine everywhere without issues.

[SviridoffA]

Forgot to add that I use Ubuntu 24.04 and ACS 4.19.3

[prashanthr2]

@SviridoffA You can start looking at below logs to start with.

/var/log/cloudstack/management/management-server.log from Management servers
/var/log/cloud.log from the console proxy VM

Grep the above logs with console or better trace through the logs during the time of the issue. I can take a look at them as well if you can send them across.

[weizhouapache]

@SviridoffA
are the VMs new or running for long period (for example 1 year) ?

[SviridoffA]

@weizhouapache
This specific virtual machine is about a year old, but before my actions to fix this issue (manual migration), the problem was observed on all virtual machines of these 4 hosts, including new VMs. These 4 nodes are the oldest in the entire infrastructure.
UPD: I also checked the command history and found that not long ago this cluster was expanded by two more nodes. The new nodes, during deployment, pulled in more recent versions of linstor and drbd, and I decided to update via install --only-upgrade, after which a sequential reboot of these 4 nodes was performed.

[SviridoffA]

The issues were not observed on the two new nodes of this cluster.

[weizhouapache]

The issues were not observed on the two new nodes of this cluster.

I suspect you are facing the issue #9718

[SviridoffA]

@weizhouapache Its realy looks similar, How can I check versions of certificates? just to be sure this is the same problem. I have checked creation date of this nodes and its 14 Oct 2024.

[SviridoffA]

Guys, thank you all for your help!
This is not a simple task (for me:) ) to figure out which certificate QEMU process is running with, and I decided not to bother with this task. All other indirect factors indicate that @weizhouapache is right, and it's also reliably known about the issue of launching virtual machines with an outdated certificate when the VMs were started before the certificate renewal. I think we can close this discussion.
My current certificates:

root@node2:~# openssl x509 -in /etc/cloudstack/agent/cloud.crt -noout -dates
notBefore=Sep 28 23:35:22 2025 GMT
notAfter=Sep 29 11:35:22 2026 GMT
root@node2:~# openssl x509 -in /etc/pki/libvirt/servercert.pem -noout -dates
notBefore=Sep 28 23:35:22 2025 GMT
notAfter=Sep 29 11:35:22 2026 GMT

As far as I understand, if auto-renewal is enabled, the certificates will be regenerated 15 days before expiration. In that case, everything adds up.
I set the CA framework cert validity period to 3650 days, so next year I will encounter this issue again, after which certificates with a 10-year term will be issued.

[weizhouapache]

thanks for the update @SviridoffA
for existing running vms, you can migrate or stop/start them