Failed to add static route. #12672

olegivanovicholegivanovich-ux · 2026-02-20T09:39:30Z

olegivanovicholegivanovich-ux
Feb 20, 2026

Hello,

We are using CloudStack version 4.22 with Router version 4.22.0 and have encountered an issue when creating static routes within a VPC.

When selecting a specific Private Gateway, we are able to specify a static route with only one parameter — the Destination Network CIDR. However, when attempting to create the route, we receive an error.

Could you please assist us in troubleshooting this issue?

Thank you in advance for your support.

Managemen-server.log

2026-02-20 12:25:36,736 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Executing AsyncJob {"accountId":90,"cmd":"org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"115\",\"sessionkey\":\"xr_OMkDr6EUkcvqm1uBU28Nyn18\",\"httpmethod\":\"POST\",\"cidr\":\"10.24.192.0\/24\",\"ctxStartEventId\":\"2640380\",\"id\":\"518\",\"ctxDetails\":\"{\\\"interface com.cloud.network.vpc.StaticRoute\\\":\\\"cc63282f-b3eb-4856-b927-abe32c7476fb\\\",\\\"interface com.cloud.network.vpc.VpcGateway\\\":\\\"39341bcd-6398-45c4-8ef1-cec0e7ce6630\\\"}\",\"ctxAccountId\":\"90\",\"uuid\":\"cc63282f-b3eb-4856-b927-abe32c7476fb\",\"gatewayid\":\"39341bcd-6398-45c4-8ef1-cec0e7ce6630\",\"cmdEventType\":\"STATIC.ROUTE.CREATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":143465,"initMsid":108597816265214,"instanceId":518,"instanceType":"StaticRoute","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":115,"uuid":"a132954b-59ac-4c0e-9af5-9df3481538c6"}
2026-02-20 12:25:36,752 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Found 6 static routes for VPC 69
2026-02-20 12:25:36,763 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Applying static routes for vpc VPC {"id":69,"name":"becloud_askue","uuid":"0045da96-4455-42e0-b12a-5f5c26f5d63a"}
2026-02-20 12:25:36,765 DEBUG [o.a.c.n.t.BasicNetworkTopology] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) APPLYING STATIC ROUTES RULES
2026-02-20 12:25:36,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Wait time setting on com.cloud.agent.api.routing.SetStaticRouteCommand is 1800 seconds
2026-02-20 12:25:36,771 DEBUG [c.c.a.m.ClusteredDirectAgentAttache] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Seq 46-8219632269905127317: Routed from 108597816265214
2026-02-20 12:25:36,772 DEBUG [c.c.a.t.Request] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Seq 7-8219632269905127317: Sending  { Cmd , MgmtId: 108597816265214, via: 7(xcp-sr121-u0112-s10.aisrp.local), Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":"38","uuid":"b8b380e7-fcaf-49b9-b5b5-1b1e9f0fd903","targetCidr":"192.168.112.0/20","accountId":"107","domainId":"1","gatewayId":"14","state":"Active","vpcId":"69","vlanTag":"268","gateway":"192.168.128.1","netmask":"255.255.255.0","ipAddress":"192.168.128.100"},{"id":"44","uuid":"bebda465-ecbc-4c36-830e-8903e6f15819","targetCidr":"10.25.143.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"45","uuid":"a8306d5b-c74c-4324-b6be-f3a1c17b79fb","targetCidr":"10.25.131.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"56","uuid":"ae567124-480d-43bf-bcf6-0f3edea5deb8","targetCidr":"10.69.52.0/29","accountId":"107","domainId":"1","gatewayId":"14","state":"Active","vpcId":"69","vlanTag":"268","gateway":"192.168.128.1","netmask":"255.255.255.0","ipAddress":"192.168.128.100"},{"id":"111","uuid":"cdb1370e-f426-475c-a786-fb9a82cc92e8","targetCidr":"10.24.191.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"518","uuid":"cc63282f-b3eb-4856-b927-abe32c7476fb","targetCidr":"10.24.192.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Add","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"}],"accessDetails":{"router.name":"r-8907-VM","router.ip":"169.254.105.84","zone.network.type":"Advanced"},"wait":"0","bypassHostMaintenance":"false"}}] }
2026-02-20 12:25:36,773 DEBUG [c.c.a.t.Request] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Seq 7-8219632269905127317: Executing:  { Cmd , MgmtId: 108597816265214, via: 7(xcp-sr121-u0112-s10.aisrp.local), Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":"38","uuid":"b8b380e7-fcaf-49b9-b5b5-1b1e9f0fd903","targetCidr":"192.168.112.0/20","accountId":"107","domainId":"1","gatewayId":"14","state":"Active","vpcId":"69","vlanTag":"268","gateway":"192.168.128.1","netmask":"255.255.255.0","ipAddress":"192.168.128.100"},{"id":"44","uuid":"bebda465-ecbc-4c36-830e-8903e6f15819","targetCidr":"10.25.143.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"45","uuid":"a8306d5b-c74c-4324-b6be-f3a1c17b79fb","targetCidr":"10.25.131.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"56","uuid":"ae567124-480d-43bf-bcf6-0f3edea5deb8","targetCidr":"10.69.52.0/29","accountId":"107","domainId":"1","gatewayId":"14","state":"Active","vpcId":"69","vlanTag":"268","gateway":"192.168.128.1","netmask":"255.255.255.0","ipAddress":"192.168.128.100"},{"id":"111","uuid":"cdb1370e-f426-475c-a786-fb9a82cc92e8","targetCidr":"10.24.191.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Active","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"},{"id":"518","uuid":"cc63282f-b3eb-4856-b927-abe32c7476fb","targetCidr":"10.24.192.0/24","accountId":"107","domainId":"1","gatewayId":"18","state":"Add","vpcId":"69","vlanTag":"vlan://280","gateway":"10.69.113.38","netmask":"255.255.255.252","ipAddress":"10.69.113.37"}],"accessDetails":{"router.name":"r-8907-VM","router.ip":"169.254.105.84","zone.network.type":"Advanced"},"wait":"0","bypassHostMaintenance":"false"}}] }
2026-02-20 12:25:36,773 DEBUG [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Transforming com.cloud.agent.api.routing.SetStaticRouteCommand to ConfigItems
2026-02-20 12:25:36,773 DEBUG [c.c.a.r.v.f.SetStaticRouteConfigItem] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Transformed filename: static_routes.json to: static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c
2026-02-20 12:25:36,776 DEBUG [c.c.h.x.r.XenServer650Resource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Copying VR with ip 169.254.105.84 config file into host 10.69.104.73
2026-02-20 12:25:37,493 DEBUG [c.c.h.x.r.XenServer650Resource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) VR Config file static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c got created in VR, IP: 169.254.105.84 with content
2026-02-20 12:25:37,493 DEBUG [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Processing FileConfigItem, copying 661 characters to static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c took 720ms
2026-02-20 12:25:37,493 DEBUG [c.c.h.x.r.XenServer650Resource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Executing command in VR: /opt/cloud/bin/router_proxy.sh update_config.py 169.254.105.84 static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c
2026-02-20 12:25:38,159 ERROR [c.c.u.s.SshHelper] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) SSH execution of command /opt/cloud/bin/router_proxy.sh update_config.py 169.254.105.84 static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c has an error status code in return. Result output: Traceback (most recent call last):
2026-02-20 12:25:38,160 DEBUG [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Processing ScriptConfigItem, executing update_config.py static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c took 667ms
2026-02-20 12:25:38,160 DEBUG [c.c.a.m.D.Task] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Seq 46-8219632269905127317: Response Received:
2026-02-20 12:25:38,160 DEBUG [c.c.a.t.Request] (DirectAgent-367:[ctx-64a2d2ea]) (logid:a132954b) Seq 7-8219632269905127317: Processing:  { Ans: , MgmtId: 108597816265214, via: 7(xcp-sr121-u0112-s10.aisrp.local), Ver: v1, Flags: 0, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: ","null - failed: Traceback (most recent call last):
2026-02-20 12:25:38,160 DEBUG [c.c.a.t.Request] (API-Job-Executor-67:[ctx-75c14779, job-143465, ctx-17c5003a]) (logid:a132954b) Seq 7-8219632269905127317: Received:  { Ans: , MgmtId: 108597816265214, via: 7(xcp-sr121-u0112-s10.aisrp.local), Ver: v1, Flags: 0, { GroupAnswer } }
2026-02-20 12:25:38,175 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Complete async job-143465, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create static route"}
2026-02-20 12:25:38,176 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Publish async job-143465 complete on message bus
2026-02-20 12:25:38,176 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Wake up jobs related to job-143465
2026-02-20 12:25:38,176 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Update db status for job-143465
2026-02-20 12:25:38,177 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Wake up jobs joined with job-143465 and disjoin all subjobs created from job- 143465
2026-02-20 12:25:38,183 DEBUG [c.c.a.ApiServer] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Retrieved cmdEventType from job info: STATIC.ROUTE.CREATE
2026-02-20 12:25:38,183 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Done executing org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd for job-143465
2026-02-20 12:25:38,184 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-67:[ctx-75c14779, job-143465]) (logid:a132954b) Remove job-143465 from job monitoring

cloud.log on router:

2026-02-20 09:25:37,774 INFO     update_config.py :: Processing incoming file => static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c
2026-02-20 09:25:37,774 INFO     Processing JSON file static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c
2026-02-20 09:25:37,774 INFO     Continuing with the processing of file '/var/cache/cloud/static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c'
2026-02-20 09:25:37,775 INFO     Command of type staticroutes received
2026-02-20 09:25:37,776 INFO     Executing: ip addr show dev eth0
2026-02-20 09:25:37,779 INFO     Executing: ip addr show dev eth1
2026-02-20 09:25:37,782 INFO     Executing: ip addr show dev eth2
2026-02-20 09:25:37,785 INFO     Executing: ip addr show dev eth3
2026-02-20 09:25:37,787 INFO     Executing: ip addr show dev eth4
2026-02-20 09:25:37,790 INFO     Executing: ip addr show dev eth0
2026-02-20 09:25:37,793 INFO     Address found in DataBag ==> {'add': True, 'broadcast': '169.254.255.255', 'cidr': '169.254.105.84/16', 'device': 'eth0', 'gateway': '', 'netmask': '255.255.0.0', 'network': '169.254.0.0/16', 'nic_dev_id': '0', 'nw_type': 'control', 'one_to_one_nat': False, 'public_ip': '169.254.105.84', 'size': '16', 'source_nat': False}
2026-02-20 09:25:37,793 INFO     Address 169.254.105.84/16 on device eth0 already configured
2026-02-20 09:25:37,793 INFO     Executing: ip addr show dev eth1
2026-02-20 09:25:37,796 INFO     Address found in DataBag ==> {'add': True, 'broadcast': '93.125.23.255', 'cidr': '93.125.23.187/25', 'device': 'eth1', 'first_i_p': True, 'gateway': '93.125.23.129', 'is_private_gateway': False, 'mtu': '1500', 'netmask': '255.255.255.128', 'network': '93.125.23.128/25', 'new_nic': False, 'nic_dev_id': 1, 'nw_type': 'public', 'one_to_one_nat': False, 'public_ip': '93.125.23.187', 'size': '25', 'source_nat': True, 'vif_mac_address': '1e:00:5c:00:00:48'}
2026-02-20 09:25:37,796 INFO     Address 93.125.23.187/25 on device eth1 already configured
2026-02-20 09:25:37,796 INFO     Adding route table: 101 Table_eth1 to /etc/iproute2/rt_tables if not present 
2026-02-20 09:25:37,796 INFO     Executing: ip rule show
2026-02-20 09:25:37,799 INFO     Executing: ip rule show
2026-02-20 09:25:37,801 INFO     Executing: ip link show eth1 | grep ' state '
2026-02-20 09:25:37,805 INFO     Executing2: arping -c 1 -I eth1 -A -U -s 93.125.23.187 93.125.23.129
2026-02-20 09:25:37,805 INFO     Adding route: dev eth1 table: Table_eth1 network: 93.125.23.129 if not present
2026-02-20 09:25:37,805 INFO     Executing: ip route show default via 93.125.23.129 table Table_eth1 proto static
2026-02-20 09:25:37,808 INFO     Adding route: dev eth1 table: Table_eth1 network: 172.16.80.80/28 if not present
2026-02-20 09:25:37,808 INFO     Executing: ip route show  172.16.80.80/28 table Table_eth1 proto static
2026-02-20 09:25:37,811 INFO     Executing: ip rule show
2026-02-20 09:25:37,814 INFO     Adding route: dev eth1 table: Table_eth1 network: 93.125.23.128/25 if not present
2026-02-20 09:25:37,814 INFO     Executing: ip route show  93.125.23.128/25 table Table_eth1 proto static
2026-02-20 09:25:37,817 INFO     Executing: sudo ip route flush cache
2026-02-20 09:25:37,827 INFO     Checking if default ipv4 route is present
2026-02-20 09:25:37,827 INFO     Executing: ip -4 route list 0/0
2026-02-20 09:25:37,830 INFO     Default route found: default via 93.125.23.129 dev eth1 
2026-02-20 09:25:37,830 INFO     Address found in DataBag ==> {'add': True, 'broadcast': '93.125.23.255', 'cidr': '93.125.23.188/25', 'device': 'eth1', 'first_i_p': True, 'gateway': '93.125.23.129', 'is_private_gateway': False, 'mtu': '1500', 'netmask': '255.255.255.128', 'network': '93.125.23.128/25', 'new_nic': False, 'nic_dev_id': 1, 'nw_type': 'public', 'one_to_one_nat': False, 'public_ip': '93.125.23.188', 'size': '25', 'source_nat': False, 'vif_mac_address': '1e:00:5c:00:00:48'}
2026-02-20 09:25:37,830 INFO     Address 93.125.23.188/25 on device eth1 already configured
2026-02-20 09:25:37,830 INFO     Adding route table: 101 Table_eth1 to /etc/iproute2/rt_tables if not present 
2026-02-20 09:25:37,830 INFO     Executing: ip rule show
2026-02-20 09:25:37,833 INFO     Executing: ip rule show
2026-02-20 09:25:37,836 INFO     Executing: ip link show eth1 | grep ' state '
2026-02-20 09:25:37,839 INFO     Executing2: arping -c 1 -I eth1 -A -U -s 93.125.23.188 93.125.23.129
2026-02-20 09:25:37,840 INFO     Adding route: dev eth1 table: Table_eth1 network: 93.125.23.129 if not present
2026-02-20 09:25:37,840 INFO     Executing: ip route show default via 93.125.23.129 table Table_eth1 proto static
2026-02-20 09:25:37,843 INFO     Adding route: dev eth1 table: Table_eth1 network: 172.16.80.80/28 if not present
2026-02-20 09:25:37,843 INFO     Executing: ip route show  172.16.80.80/28 table Table_eth1 proto static
2026-02-20 09:25:37,846 INFO     Executing: ip rule show
2026-02-20 09:25:37,848 INFO     Adding route: dev eth1 table: Table_eth1 network: 93.125.23.128/25 if not present
2026-02-20 09:25:37,848 INFO     Executing: ip route show  93.125.23.128/25 table Table_eth1 proto static
2026-02-20 09:25:37,851 INFO     Executing: sudo ip route flush cache
2026-02-20 09:25:37,860 INFO     Checking if default ipv4 route is present
2026-02-20 09:25:37,860 INFO     Executing: ip -4 route list 0/0
2026-02-20 09:25:37,862 INFO     Default route found: default via 93.125.23.129 dev eth1 
2026-02-20 09:25:37,862 INFO     Executing: ip addr show dev eth2
2026-02-20 09:25:37,865 INFO     Address found in DataBag ==> {'add': True, 'broadcast': '192.168.128.255', 'cidr': '192.168.128.100/24', 'device': 'eth2', 'first_i_p': False, 'gateway': '192.168.128.1', 'is_private_gateway': True, 'mtu': '1500', 'netmask': '255.255.255.0', 'network': '192.168.128.0/24', 'new_nic': False, 'nic_dev_id': 2, 'nw_type': 'public', 'one_to_one_nat': False, 'public_ip': '192.168.128.100', 'size': '24', 'source_nat': True, 'vif_mac_address': '1e:00:4f:00:00:cc'}
2026-02-20 09:25:37,865 INFO     Address 192.168.128.100/24 on device eth2 already configured
2026-02-20 09:25:37,865 INFO     Adding route table: 102 Table_eth2 to /etc/iproute2/rt_tables if not present 
2026-02-20 09:25:37,865 INFO     Executing: ip rule show
2026-02-20 09:25:37,868 INFO     Executing: ip rule show
2026-02-20 09:25:37,871 INFO     Executing: ip link show eth2 | grep ' state '
2026-02-20 09:25:37,874 INFO     Executing2: arping -c 1 -I eth2 -A -U -s 192.168.128.100 192.168.128.1
2026-02-20 09:25:37,875 INFO     Adding route: dev eth2 table: Table_eth2 network: 192.168.128.1 if not present
2026-02-20 09:25:37,875 INFO     Executing: ip route show default via 192.168.128.1 table Table_eth2 proto static
2026-02-20 09:25:37,878 INFO     Adding route: dev eth2 table: Table_eth2 network: 172.16.80.80/28 if not present
2026-02-20 09:25:37,878 INFO     Executing: ip route show  172.16.80.80/28 table Table_eth2 proto static
2026-02-20 09:25:37,880 INFO     Executing: ip rule show
2026-02-20 09:25:37,883 INFO     Adding route: dev eth2 table: Table_eth2 network: 192.168.128.0/24 if not present
2026-02-20 09:25:37,883 INFO     Executing: ip route show  192.168.128.0/24 table Table_eth2 proto static
2026-02-20 09:25:37,886 INFO     Executing: sudo ip route flush cache

Error in executing update_config.py

root@r-8907-VM:~# zcat /var/cache/cloud/processed/static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c.gz > /var/cache/cloud/static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c

root@r-8907-VM:~# cat /var/cache/cloud/static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c 
{"routes":[{"revoke":false,"ip_address":"192.168.128.100","gateway":"192.168.128.1","network":"192.168.112.0/20"},{"revoke":false,"ip_address":"10.69.113.37","gateway":"10.69.113.38","network":"10.25.143.0/24"},{"revoke":false,"ip_address":"10.69.113.37","gateway":"10.69.113.38","network":"10.25.131.0/24"},{"revoke":false,"ip_address":"192.168.128.100","gateway":"192.168.128.1","network":"10.69.52.0/29"},{"revoke":false,"ip_address":"10.69.113.37","gateway":"10.69.113.38","network":"10.24.191.0/24"},{"revoke":false,"ip_address":"10.69.113.37","gateway":"10.69.113.38","network":"10.24.192.0/24"}],"type":"staticroutes","delete_from_processed_cache":false}root@r-8907-VM:~# 

root@r-8907-VM:~# /opt/cloud/bin/update_config.py static_routes.json.7d215085-5ad5-4da6-9bdd-388f1c70050c
Traceback (most recent call last):
  File "/opt/cloud/bin/update_config.py", line 147, in <module>
    process_file()
  File "/opt/cloud/bin/update_config.py", line 57, in process_file
    finish_config()
  File "/opt/cloud/bin/update_config.py", line 42, in finish_config
    returncode = configure.main(sys.argv)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloud/bin/configure.py", line 1647, in main
    config.address().process()
  File "/opt/cloud/bin/cs/CsAddress.py", line 132, in process
    ip.post_configure(address)
  File "/opt/cloud/bin/cs/CsAddress.py", line 370, in post_configure
    self.post_config_change("add")
  File "/opt/cloud/bin/cs/CsAddress.py", line 824, in post_config_change
    self.fw_vpcrouter()
  File "/opt/cloud/bin/cs/CsAddress.py", line 606, in fw_vpcrouter
    if static_route['ip_address'] == self.address['public_ip'] and not static_route['revoke']:
       ~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'ip_address'
root@r-8907-VM:~#

Answered by weizhouapache

Feb 20, 2026

@olegivanovicholegivanovich-ux
#11967 is not included in 4.22.0.0, it will be included in 4.22.1.0
as a workaround, you can refer to #11967 (comment)

View full answer

weizhouapache · 2026-02-20T09:51:56Z

weizhouapache
Feb 20, 2026
Collaborator

@olegivanovicholegivanovich-ux
can you share the content of /etc/cloudstack/staticroutes.json ?

2 replies

olegivanovicholegivanovich-ux Feb 20, 2026
Author

root@r-8907-VM:~# cat /etc/cloudstack/staticroutes.json
{
"10.24.191.0/24": {
"gateway": "10.69.113.38",
"ip_address": "10.69.113.37",
"network": "10.24.191.0/24",
"revoke": false
},
"10.24.192.0/24": {
"gateway": "10.69.113.38",
"ip_address": "10.69.113.37",
"network": "10.24.192.0/24",
"revoke": false
},
"10.25.131.0/24": {
"gateway": "10.69.113.38",
"ip_address": "10.69.113.37",
"network": "10.25.131.0/24",
"revoke": false
},
"10.25.143.0/24": {
"gateway": "10.69.113.38",
"ip_address": "10.69.113.37",
"network": "10.25.143.0/24",
"revoke": false
},
"10.69.52.0/29": {
"gateway": "192.168.128.1",
"ip_address": "192.168.128.100",
"network": "10.69.52.0/29",
"revoke": false
},
"172.16.20.0/24": {
"gateway": "192.168.128.100",
"network": "172.16.20.0/24",
"revoke": false
},
"172.16.80.178/32": {
"gateway": "192.168.128.1",
"ip_address": "192.168.128.100",
"network": "172.16.80.178/32",
"revoke": false
},
"192.168.112.0/20": {
"gateway": "192.168.128.1",
"ip_address": "192.168.128.100",
"network": "192.168.112.0/20",
"revoke": false
},
"id": "staticroutes"

weizhouapache Feb 20, 2026
Collaborator

"172.16.20.0/24": {
"gateway": "192.168.128.100",
"network": "172.16.20.0/24",
"revoke": false
},

@olegivanovicholegivanovich-ux
I think the issue is caused by this which misses "ip_address" ?
can you share more information of this static route ?

weizhouapache · 2026-02-20T10:01:18Z

weizhouapache
Feb 20, 2026
Collaborator

I see, it is a static route with next hop
strange, the ip address is missing in the json file
I am able to reproduce the issue

0 replies

weizhouapache · 2026-02-20T10:37:56Z

weizhouapache
Feb 20, 2026
Collaborator

@olegivanovicholegivanovich-ux
it has been fixed by #11967
can you apply the changes in the VPC VR ?

6 replies

olegivanovicholegivanovich-ux Feb 20, 2026
Author

We updated CloudStack to version 4.22 from the repository using "dnf update cloudstack-management" on December 5th. However, the bugfix was applied on December 16th and was not released as a separate branch.

As a result, both our version and yours are labeled 4.22.0, but they are effectively different. Because of this, the package manager does not recognize any available updates.

Could you please advise on how we should proceed in this situation?

olegivanovicholegivanovich-ux Feb 20, 2026
Author

We were able to resolve the issue by editing the staticroutes.json file and removing entries that were missing the ip_address field.

This allowed us to manage Static Routes within the Private Gateway. However, inside the VPC, when multiple Private Gateways are present, this functionality does not work — it requires a next hop, but fails when one is provided.

weizhouapache Feb 20, 2026
Collaborator

@olegivanovicholegivanovich-ux
#11967 is not included in 4.22.0.0, it will be included in 4.22.1.0
as a workaround, you can refer to #11967 (comment)

Answer selected by olegivanovicholegivanovich-ux

olegivanovicholegivanovich-ux Feb 23, 2026
Author

Please clarify when we can expect the 4.22.1 release?
We would like to update the entire system at once.

weizhouapache Feb 23, 2026
Collaborator

ease clarify when we can expect the 4.22.1 release?
We would like to update the entire system at once.

@olegivanovicholegivanovich-ux
you can find the update at : https://lists.apache.org/thread/d0dnd6rb1hvd633hfncb60tj6h4vk0f6

olegivanovicholegivanovich-ux Feb 23, 2026
Author

Got it, thank you!

Failed to add static route. #12672

Uh oh!

olegivanovicholegivanovich-ux Feb 20, 2026

Replies: 3 comments · 8 replies

Uh oh!

weizhouapache Feb 20, 2026 Collaborator

Uh oh!

olegivanovicholegivanovich-ux Feb 20, 2026 Author

Uh oh!

weizhouapache Feb 20, 2026 Collaborator

Uh oh!

weizhouapache Feb 20, 2026 Collaborator

Uh oh!

weizhouapache Feb 20, 2026 Collaborator

Uh oh!

olegivanovicholegivanovich-ux Feb 20, 2026 Author

Uh oh!

olegivanovicholegivanovich-ux Feb 20, 2026 Author

Uh oh!

weizhouapache Feb 20, 2026 Collaborator

Uh oh!

olegivanovicholegivanovich-ux Feb 23, 2026 Author

Uh oh!

weizhouapache Feb 23, 2026 Collaborator

Uh oh!

olegivanovicholegivanovich-ux Feb 23, 2026 Author

olegivanovicholegivanovich-ux
Feb 20, 2026

Replies: 3 comments 8 replies

weizhouapache
Feb 20, 2026
Collaborator

olegivanovicholegivanovich-ux Feb 20, 2026
Author

weizhouapache Feb 20, 2026
Collaborator

weizhouapache
Feb 20, 2026
Collaborator

weizhouapache
Feb 20, 2026
Collaborator

olegivanovicholegivanovich-ux Feb 20, 2026
Author

olegivanovicholegivanovich-ux Feb 20, 2026
Author

weizhouapache Feb 20, 2026
Collaborator

olegivanovicholegivanovich-ux Feb 23, 2026
Author

weizhouapache Feb 23, 2026
Collaborator

olegivanovicholegivanovich-ux Feb 23, 2026
Author