Recurring Instance Creation Failures in CloudStack 4.19.3 on XCP-ng 8.2.1 – Management Service Restart Required

[joan-911]

Environment Details:

CloudStack Version: 4.19.3
Hypervisor: XCP-ng 8.2.1
Primary Storage: Pre-configured iSCSI on XCP-ng

Issue Observed:
All instance creation requests are timing out, and new instances fail to deploy. This problem tends to recur every 5–6 days. Once it occurs, restarting the management service temporarily resolves the issue.

We would like to know if anyone has faced a similar issue in ACS with XCP-ng. Are there any configuration tweaks available to prevent having to restart the management service each time?

Additional Information:
Logs related to disk creation are attached for reference.

2025-10-22 10:24:33,667 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Forwarding Seq 20-6844627008673079170:  { Cmd , MgmtId: 271367514881978, via: 20(CLD-Z1-XCPH01), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"9f1a2c02-4c9b-4910-b3e8-c1026e42addf","origUrl":"http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2","uuid":"1036c354-65b7-492f-96fc-848ee19f49ab","id":"380","format":"VHD","accountId":"6","checksum":"{SHA-512}e7df5f0fc04318bf4bf2b28fa0e630cfb3b7cd3f783202382202c8404fc8deaadc5c03e8b0c2137a2fb7d998ee3901e3d94f280587999fee6659399c064b59ea","hvm":"true","displayText":"systemvm-xenserver-4.19.1","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","size":"(4.88 GB) 5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2cf5fcb9-9570-4263-95ea-98004c0efa98","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"ROOT-15673","size":"(4.88 GB) 5242880000","volumeId":"6223","vmName":"r-15673-VM","accountId":"1028","format":"VHD","provisioningType":"THIN","poolId":"7","id":"6223","deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}] } to 108462341209969
2025-10-22 10:54:33,668 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Waiting some more time because this is the current command
2025-10-22 11:24:33,668 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Waiting some more time because this is the current command
2025-10-22 11:24:33,674 WARN  [c.c.a.m.AgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Timed out on Seq 20-6844627008673079170:  { Cmd , MgmtId: 271367514881978, via: 20(CLD-Z1-XCPH01), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"9f1a2c02-4c9b-4910-b3e8-c1026e42addf","origUrl":"http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2","uuid":"1036c354-65b7-492f-96fc-848ee19f49ab","id":"380","format":"VHD","accountId":"6","checksum":"{SHA-512}e7df5f0fc04318bf4bf2b28fa0e630cfb3b7cd3f783202382202c8404fc8deaadc5c03e8b0c2137a2fb7d998ee3901e3d94f280587999fee6659399c064b59ea","hvm":"true","displayText":"systemvm-xenserver-4.19.1","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","size":"(4.88 GB) 5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2cf5fcb9-9570-4263-95ea-98004c0efa98","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"ROOT-15673","size":"(4.88 GB) 5242880000","volumeId":"6223","vmName":"r-15673-VM","accountId":"1028","format":"VHD","provisioningType":"THIN","poolId":"7","id":"6223","deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-10-22 11:24:33,674 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Forwarding to peer to cancel due to timeout
2025-10-22 11:24:33,675 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Seq 20-6844627008673079170: Cancelling.
2025-10-22 11:24:33,675 DEBUG [o.a.c.s.RemoteHostEndPoint] (Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497) (logid:13a10f8a) Failed to send command, due to Agent:20, com.cloud.exception.OperationTimedoutException: Commands 6844627008673079170 to Host 20 timed out after 3600
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:20, com.cloud.exception.OperationTimedoutException: Commands 6844627008673079170 to Host 20 timed out after 3600
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:20, com.cloud.exception.OperationTimedoutException: Commands 6844627008673079170 to Host 20 timed out after 3600  

[DaanHoogland]

@joan-911 , this looks like a storage issue and not in the management server.
I know there are some iscsi issues (most of them solved). Have a look at iscsi related issues
It might of course be that some storage issue is causing the agent (which is embedded in the management server for xen-likes) to be corupted.

• Is the degration really regular?
• does it coincide with any heavy storage jobs (backup/db cleanup/coalesce)?
  It might also pay to look for errors before this in the management server logs.

[joan-911]

Thanks for your observations, @DaanHoogland.
We’ve been experiencing this degradation consistently for the past few weeks, which has forced us to restart the Management Service every few days.

We are aware of some iSCSI performance issues in our infrastructure and are working on optimizing the storage layer. I’ll review the management logs further to look for additional clues.

[rajujith]

@joan-911 find the template copy operation details in the other management server with the 'Seq 20-6844627008673079170', also check the xensource.log and the SMlog on the XCP-ng to find more on the error.