wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host by wido · Pull Request #13758 · apache/cloudstack

wido · 2026-07-31T12:04:12Z

Description

This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent link-local gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.

Direct Routed networks live on their own physical network, carrying the new ROUTED isolation method. This is what makes the feature adoptable in existing environments: an operator adds one physical network to an existing zone and gains L3 networks next to everything already running there — no changes to existing physical networks, offerings or Instances.

IPv6-only networks are fully supported. Nothing in this network type depends on IPv4 (no DHCP, no password/metadata service), so each address family is optional: IPv4 is a subnet (cidr=2.57.59.0/24, or netmask + start/end IP), IPv6 is an ip6cidr alone — IPv6 addresses are computed from the subnet and the NIC MAC with EUI-64, so no range and no pool exist. No gateways are declared or stored for either family, saving those addresses; Instances always use the shared link-local gateway.

Management server

GuestType.L3; the guest_type column is char(32), so no schema change. BroadcastDomainType.Routed: each network carries a routed://<id> broadcast domain whose id names the per-network bridge on every host — allocated from the ROUTED physical network's vnet range, or chosen by the operator (specifyVlan offerings) so bridge names are plannable before the network exists.
Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared, including the deploy-time gate), Dhcp rejected as not supported and not needed. Network mode, specifyVlan-as-required and VPC use are rejected. A DefaultL3NetworkOffering is created on install and upgrade.
DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. After allocation the NicProfile is forced into host-route form (/32 or /128 + link-local gateway), which is also the signature by which ConfigDrive recognises these NICs.
Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide.
SystemVMs work on routed public IP ranges (createVlanIpRange with vlan=routed://<id>): their public NICs take the same host-route form, with the IPv6 address derived EUI-64 from the range's subnet.

ConfigDrive

Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) would leave these NICs with no addressing at all.
A direct routed IPv4 network emits a network-level gateway key instead of a default-route entry: cloud-init derives the required on-link flag only from that key (netplan renderer since 23.1, networkd since 24.2), never from routes-list entries. This sets the guest requirement: cloud-init >= 23.1 with netplan, or >= 24.2 with networkd.

KVM agent

One uplink-less bridge per network, brdr-<routed id>, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding, strict rp_filter, and a MAC derived deterministically from the routed id — identical on every hypervisor, so a live-migrated Instance's ARP/NDP cache stays valid and cutover is seamless. Separate bridges make isolation between networks topological rather than a filtering concern.
BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host routes (v4 and v6), regardless of the host-wide EVPN property, whose meaning is unchanged.
security_group.py gains a --directrouted dispatch: rules for these Instances match by ipset destination rather than bridge port on the return path. Because a routed packet between two Instances on one host enters on one brdr bridge and leaves on another, the L3 framework runs two passes from a single FORWARD hook (shared BF-L3 chains): the source Instance's egress rules mark the packet, the destination Instance's ingress rules give the verdict, so both are always evaluated regardless of bridge creation order. A golden-file test harness pins the classic (bridged) rule stream byte-for-byte, proving existing deployments are untouched, and walks packets through the generated L3 rules.
Host protection: an Instance's packets enter the host's own IP stack, so modifybrdr.sh installs once per host an INPUT chain for brdr-+ that admits only conntrack replies, ICMP echo and IPv6 neighbour discovery, plus an ip6tables rpfilter rule as the IPv6 counterpart of rp_filter=1. Forwarding towards management and storage subnets remains the operator's host firewall policy, like the routing daemon.

root@hv-138-d03-22:~# ip addr show dev brdr-578
15: brdr-578:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:00:00:00:02:42 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.1/32 scope global brdr-578
       valid_lft forever preferred_lft forever
    inet6 fe80::c00:ff:fe00:242/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link
       valid_lft forever preferred_lft forever
root@hv-138-d03-22:~#

Other changes this PR makes to CloudStack

Fixes and improvements that ride along because the feature exposed them, but apply generally:

createNetwork gains an optional cidr parameter (L3-only for now, purely additive).
Auto-allocated ids are exempt from the "VLAN tag is already being used for dynamic vlan allocation" check in createVlanAndPublicIpRange — previously Shared networks were exempted one layer up but not there.
A failed VM start (uncaught RuntimeException) and a failed PrepareForMigration on the destination now unplug the NICs they plugged; previously bridges (and here: host routes) leaked.
canUseForDeploy() treats Shared and L3 alike and counts the real IPv4 pool; IPv6-only networks are deployable from the UI wizard.
The zone-wide IPv6 overlap check keys on ip6_cidr instead of ip6_gateway (what it actually compares).
UI: L3 network creation form, ROUTED in the zone wizard and physical network forms, and the Instances list shows active IPv4/IPv6 addresses — IPv6 shortened with the tail visible, click copies the full address.
SystemVMs (CPVM/SSVM) receive their IPv6 address and gateway as boot arguments, as the VR already did, and common.sh installs the IPv6 default route statically from ip6gateway instead of relying on a router advertisement. This also applies to classic VLAN public networks with IPv6, where systemvms previously had no IPv6 default route unless a fabric router sent RAs.
The design document's §15 lists review findings that were consciously deferred (failed-migration rollback on the destination, root-admin-only network creation, KVM-only enforcement, vlan range CRUD on gateway-less rows, and others) so they are tracked rather than forgotten.

The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.

This implements issue #12210

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

The feature was tested end to end on real hardware in a datacenter in September 2026: a KVM zone with a ROUTED physical network, SystemVMs on a routed public range, dual-stack and IPv6-only L3 networks, Instances with security groups, and live migration between hosts. Evidence of all tests is recorded, and the lab is available for remote access on request for anyone who wants to validate the feature without building their own environment.

Verified guest behaviour: Ubuntu 26.04 with cloud-init (26.1) and ConfigDrive as the datasource works as expected — IPv4 default route installed with on-link, EUI-64 IPv6 configured, dual-stack and IPv6-only.

Unit tests cover the guru, offering validation, ConfigDrive generation, the deploy gates and the address-family validation; the security_group.py golden harness pins the classic rule stream; integration tests (test_l3_networks.py) cover creation (subnet forms, operator-chosen ids, IPv6-only, gateway-ignoring, overlap and incomplete-family rejection) and deployment.

Pending lab validation for the review follow-up commits (September 2026): the two-pass security-group rules (cross-account Instance-to-Instance traffic on one host, iptables -C idempotency of the MARK rules), the host protection chain (Instances must still resolve and ping 169.254.0.1 / fe80::1) and the new unplug order. Design doc §12.2 and §13 list exactly what to check.

codecov · 2026-07-31T12:13:34Z

Codecov Report

❌ Patch coverage is 49.40375% with 297 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.93%. Comparing base (a723d44) to head (a983c61).
⚠️ Report is 48 commits behind head on main.

Files with missing lines	Patch %	Lines
...ain/java/com/cloud/network/NetworkServiceImpl.java	43.58%	49 Missing and 17 partials ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	4.08%	47 Missing ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	49.36%	38 Missing and 2 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	2.85%	33 Missing and 1 partial ⚠️
...om/cloud/network/guru/DirectRoutedNetworkGuru.java	54.41%	18 Missing and 13 partials ⚠️
...cloud/hypervisor/kvm/resource/BridgeVifDriver.java	80.80%	11 Missing and 8 partials ⚠️
...oud/network/security/SecurityGroupManagerImpl.java	0.00%	10 Missing ⚠️
...ud/agent/api/NetworkRulesVmSecondaryIpCommand.java	27.27%	8 Missing ⚠️
...tack/api/command/user/vm/RemoveIpFromVmNicCmd.java	0.00%	5 Missing ⚠️
...m/resource/wrapper/LibvirtStartCommandWrapper.java	0.00%	5 Missing ⚠️
... and 13 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13758      +/-   ##
============================================
+ Coverage     19.78%   19.93%   +0.14%     
- Complexity    19992    20263     +271     
============================================
  Files          6371     6373       +2     
  Lines        575903   577446    +1543     
  Branches      70496    70789     +293     
============================================
+ Hits         113945   115102    +1157     
- Misses       449530   449756     +226     
- Partials      12428    12588     +160

Flag	Coverage Δ
uitests	`3.69% <ø> (+0.16%)`	⬆️
unittests	`21.21% <49.40%> (+0.15%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR introduces Direct Routed (L3) guest networks where KVM hosts perform L3 routing directly to instances (no Virtual Router, no NAT, no DHCP), using ConfigDrive-only addressing and a shared link-local gateway (169.254.0.1 / fe80::1) on per-network host bridges.

Changes:

Adds GuestType.L3 with server-side offering/network validation and a new DirectRoutedNetworkGuru that forces NICs into host-route form (/32, /128) with link-local gateways.
Updates ConfigDrive generation to always emit network_data.json for direct-routed NICs even when DHCP/DNS services are not present.
Extends the KVM agent to create/manage per-network bridges (brdr-<networkId>), program host routes/neighbour entries, and adapt security-group rule plumbing for routed traffic (including secondary IP handling).

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/src/main/java/com/cloud/utils/net/NetUtils.java	Centralizes link-local gateway/CIDR constants and adds IPv6 link-local gateway getter.
ui/src/views/offering/AddNetworkOffering.vue	Adds UI support for L3 offerings and enforces L3-specific service/provider composition.
ui/src/views/network/CreateNetwork.vue	Adds an L3 network creation tab.
ui/src/views/network/CreateL3NetworkForm.vue	New UI form for creating L3 (Direct Routed) networks.
ui/public/locales/en.json	Adds UI strings for L3 offering/network creation messaging.
tools/marvin/marvin/config/test_data.py	Adds Marvin test data for L3 offerings and L3 networks.
test/integration/smoke/test_l3_networks.py	Adds smoke/integration tests covering L3 network lifecycle and validations.
server/src/test/java/com/cloud/network/guru/DirectRoutedNetworkGuruTest.java	Unit tests for the new guru selection/design and NIC host-route forcing.
server/src/test/java/com/cloud/configuration/ConfigurationManagerImplTest.java	Unit tests for L3 offering validation rules.
server/src/main/resources/META-INF/cloudstack/server-network/spring-server-network-context.xml	Registers the new `DirectRoutedNetworkGuru` bean.
server/src/main/java/com/cloud/network/security/SecurityGroupManagerImpl.java	Ensures agent commands are sent for secondary IPs on L3 networks even when SG rules aren’t applied.
server/src/main/java/com/cloud/network/NetworkServiceImpl.java	Extends network creation/IPv6 checks/VLAN handling and secondary-IP orchestration for L3.
server/src/main/java/com/cloud/network/guru/DirectRoutedNetworkGuru.java	New guru for L3 direct routed guest networks; forces /32 + /128 form and link-local gateways.
server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java	Adds L3 offering validation + zone-wide IPv4 overlap checks for L3 ranges.
scripts/vm/network/vnet/modifymacip.sh	Adds targeted delete support for specific -4/-6 addresses (secondary IP remove).
scripts/vm/network/vnet/modifybrdr.sh	New script to create/delete per-network `brdr-*` bridges with sysctls + gateway addresses.
scripts/vm/network/tests/test_security_group.py	New unit tests for classic-vs-L3 security_group.py rule streams and `--directrouted` plumbing.
scripts/vm/network/tests/golden_default_network_rules.txt	Golden output for classic rule generation (used to ensure no regression).
scripts/vm/network/tests/golden_add_fw_framework.txt	Golden output for classic FW framework creation (ensures byte-identical behavior).
scripts/vm/network/security_group.py	Adds routed-aware framework/hooks and rule generation, plus `--directrouted` CLI plumbing.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtSecurityGroupRulesCommandWrapper.java	Passes direct-routed flag when applying SG rules for the VM’s first NIC.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtNetworkRulesVmSecondaryIpCommandWrapper.java	Threads direct-routed + apply-SG booleans into secondary-IP handling.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java	Adds `--directrouted` plumbing and direct-routed secondary-IP route/neigh programming via modifymacip.sh.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/BridgeVifDriver.java	Creates per-network bridges for direct-routed NICs and ensures MAC/IP script runs for those NICs.
engine/storage/configdrive/src/test/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilderTest.java	Adds tests for direct-routed NIC detection and network_data.json generation behavior.
engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java	Always generates network_data.json for direct-routed NICs; adds NIC signature detection.
docs/design/direct-routed-networks.md	Adds comprehensive design document for Direct Routed networks.
core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java	Adds `directRouted` and `applySecurityGroupRules` flags for secondary-IP commands.
api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java	Sends agent update for secondary IP removal on L3 even when zone SG is disabled.
api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java	Allows specifying physical network ID for L3 networks.
api/src/main/java/com/cloud/network/Network.java	Adds `GuestType.L3` and parsing support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        Network network = _networkModel.getNetwork(nic.getNetworkId());
+
+        // On a Direct Routed network the host needs a route and a static neighbour entry for the
+        // secondary IP before it is reachable at all. That is independent of security groups,
+        // which are optional there and which the Instance may not be using, so the agent is told
+        // either way - otherwise the address would stay dark until the Instance was restarted.
+        boolean directRouted = Network.GuestType.L3.equals(network.getGuestType());
+


kiranchavala · 2026-08-03T05:18:41Z

@blueorangutan package

blueorangutan · 2026-08-03T05:20:04Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-08-03T05:27:52Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18739

DaanHoogland · 2026-09-01T15:02:34Z

@wido this is still marked as wip. what is the status?

wido · 2026-09-01T15:31:13Z

@wido this is still marked as wip. what is the status?

The status is that I have real hardware in a datacenter now where I’m going to deploy this code for further testing. Please keep it WIP for now as I want to validate it there.

Review feedback on apache#13758: the multi-line inline comments narrating the code are not helpful there. Where a comment documents a real contract (guru design shape, bridge creation/naming, the MAC-before-IPv6 ordering, ConfigDrive as the only addressing channel, secondary-IP delivery independent of security groups, the link-local gateway constants) it now lives as Javadoc on the class, method or field it describes. Mid-method narration is removed; upstream one-line comments that this branch merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) are kept in their original single-line form. No code changes.

wido · 2026-09-07T10:48:31Z

@DaanHoogland fair point on the inline comments — addressed in a911e9c. The multi-line inline blocks are gone throughout the PR: where a comment documented an actual contract it moved into Javadoc on the class/method/field it describes, the mid-method narration is simply removed, and upstream one-liners this PR merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) kept their original single-line form. That covers the three spots you flagged in ConfigurationManagerImpl and CreateL3NetworkForm.vue plus the equivalent blocks elsewhere.

Review feedback on apache#13758: the multi-line inline comments narrating the code are not helpful there. Where a comment documents a real contract (guru design shape, bridge creation/naming, the MAC-before-IPv6 ordering, ConfigDrive as the only addressing channel, secondary-IP delivery independent of security groups, the link-local gateway constants) it now lives as Javadoc on the class, method or field it describes. Mid-method narration is removed; upstream one-line comments that this branch merely extended (mandatory start/end IP, vlan-creation cases, vnet allocation/release) are kept in their original single-line form. No code changes.

wido · 2026-09-07T14:30:50Z

LAB environment is working ( will share all LAB details and notes when finished ) and I have the first SystemVMs running.

root@v-28-VM:~# ip route get 9.9.9.9
9.9.9.9 via 169.254.0.1 dev eth2 src 2.57.59.69 uid 0
    cache
root@v-28-VM:~# ip -6 route get 2001:db8::1
2001:db8::1 from :: via fe80::1 dev eth2 src 2a00:f10:402:4:1c01:76ff:fe00:93 metric 1024 pref medium
root@v-28-VM:~#

root@v-28-VM:~# ip addr show dev eth2
4: eth2:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1e:01:76:00:00:93 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 2.57.59.69/32 brd 2.57.59.69 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 2a00:f10:402:4:1c01:76ff:fe00:93/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1c01:76ff:fe00:93/64 scope link
       valid_lft forever preferred_lft forever
root@v-28-VM:~#

Notice the /32 and /128 subnetmask on both IPv4 and IPv6 and the gateways on fe80::1 and 169.254.0.1

It really runs on CloudStack:

root@v-28-VM:~# dmidecode memory|grep CloudStack
        Product Name: CloudStack KVM Hypervisor
root@v-28-VM:~#

Aiming to have this in version 24.0 and give a demo/talk during CCC 2026.

… Instances Add a guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent link-local gateway (169.254.0.1 and fe80::1) that every hypervisor carries. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack. The networks live on a dedicated physical network with the new ROUTED isolation method, so the feature can be added to existing zones without touching anything already running there. Each network carries a routed://<id> broadcast domain - the id allocated from the physical network's vnet range or chosen by the operator - naming the network's uplink-less bridge (brdr-<id>) that the new modifybrdr.sh manages on every host: gateway addresses, forwarding sysctls, strict rp_filter, and a MAC derived from the routed id so it is identical on every hypervisor and live migration never invalidates the guest's neighbour cache. The existing modifymacip.sh installs the per-address host route and static neighbour entry when a NIC is plugged. Both address families are optional and IPv6-only networks are supported: IPv4 is a subnet (cidr=..., or netmask + start/end IP), IPv6 is an ip6cidr alone - addresses derive from the subnet and the NIC MAC with EUI-64, so no range exists. No gateways are declared or stored; createNetwork gains an optional cidr parameter (L3-only, additive). Security groups work through a --directrouted dispatch in security_group.py that matches the return path by ipset destination instead of bridge port; a golden-file harness pins the classic rule stream to prove existing deployments unchanged. SystemVMs run on routed public ranges (createVlanIpRange with vlan=routed://<id>) in the same host-route form. ConfigDrive network data is always generated for these NICs, with the IPv4 default route emitted as a network-level gateway key so cloud-init (netplan >= 23.1, networkd >= 24.2) renders it on-link. A DefaultL3NetworkOffering (UserData and DNS via ConfigDrive, security groups) is created on install and upgrade, and the UI gains an L3 creation form, ROUTED in the zone wizard, and active IPv4/IPv6 addresses in the Instance list with click-to-copy. General fixes that the feature surfaced but apply beyond it: auto-allocated ids are exempt from the dynamic-vlan-range check in createVlanAndPublicIpRange; a failed VM start or migration prepare now unplugs the NICs it plugged; canUseForDeploy() counts the real IPv4 pool and no longer hides IPv4-less networks from the deploy wizard; the zone-wide IPv6 overlap check keys on ip6_cidr rather than ip6_gateway. The design document with the decision log lives in docs/design/. This implements apache#12210

…condary IPs Fixes from review of the Direct Routed (L3) network implementation, on the management server side. The zone-wide IPv4 overlap check for L3 ranges sat in the createVlanIpRange API path, which rejects non-Shared guest networks and is never taken by L3 networks. It now runs in the range-creation path commitNetwork() uses, and the vlan overlap check also considers gateway-less L3 rows and treats a subnet overlap with an L3 network as a conflict, so both directions are covered. The cidr parameter now requires an explicit startip/endip to lie inside the given subnet, and rejects endip without startip; before, the stored subnet was silently re-derived from startip. Routed ids are validated and canonicalised: a positive integer of at most ten digits (brdr-<id> must fit an interface name and the bridge MAC derives from five bytes), given bare or as routed://<id>. The generic URI fallback previously accepted anything parseable. A routed public range may not take an id that a guest network holds or that lies inside a ROUTED physical network's vnet range, from which L3 networks draw auto-allocated ids. Secondary IPs on L3 NICs were unreachable because allocateSecondaryGuestIP handled only Isolated and Shared networks; L3 follows the Shared branch. Zone IPv6 DNS is no longer required for L3 networks, since DNS is optional there. Null guard for the network lookup in securityGroupRulesForVmSecIp. Inline comments this feature had touched are removed, an orphaned Javadoc is deleted, and two user-facing messages name the actual guest types.

When a direct routed NIC forced network_data.json generation, only that NIC was written. An explicit network configuration listing one interface stops cloud-init from configuring the others, so the data now covers every NIC of the Instance, matching the historical all-or-nothing semantics of the Dhcp/Dns gate.

modifybrdr.sh printed the bridge name after any ip/sysctl diagnostics and the agent read the first line of merged stdout/stderr, so a stray warning became the bridge name in the domain XML. Every operation now prints exactly one token on stdout with diagnostics on stderr, every ip and sysctl step is checked, inputs (routed id, bridge name, gateway addresses) are validated, and the agent reads the last line and validates it as an interface name. A missing modifymacip.sh is fatal for a direct routed NIC. A new -o query operation lets the agent classify a bridge without changing it, so unplug removes the Instance's routes and neighbour entries before the bridge may be deleted. Because the host routes for its Instances, their packets enter the host's own IP stack. The script installs once per host a BRDR-INPUT chain hooked with -i brdr-+ that admits only conntrack replies, ICMP echo and IPv6 neighbour discovery, and an ip6tables raw rpfilter rule as the IPv6 counterpart of rp_filter=1. Both are removed with the last brdr bridge. Null guards for the secondary IP path, and a failed security-group ipset update is reported instead of returning success.

…host For two L3 Instances on the same host the per-bridge FORWARD hooks let the first matching bridge decide terminally, so the destination's ingress rules (or the source's egress rules) were never evaluated, depending on bridge creation order. The L3 framework now uses one FORWARD hook into shared BF-L3 chains that run two passes: BF-L3-IN dispatches to the source Instance's egress rules, which mark the packet instead of accepting it, and BF-L3-OUT gives the destination Instance's terminal verdict. Unmarked packets leaving the first pass are dropped, as are packets for an L3 bridge no Instance claims; marked packets not addressed to a local L3 Instance are accepted towards the fabric. Classic bridges are unaffected and the golden files prove it. The dead rebooted-VM path loses its L3 branch, verify_network_rules expects the current rule stream, and the chain-name teardown pattern is anchored so i-2-7 no longer matches i-2-70. Tests walk packets through the generated rules for every case in both bridge creation orders.

…tworks The specifyvlan switch was hidden for L3 offerings and never sent, making operator-chosen routed ids unreachable from the UI. The L3 network form now offers admins a physical network selector limited to ROUTED physical networks and sends physicalnetworkid, takes the routed id as a number, and renders plain strings without v-html.

test_05 wrapped self.fail() inside a broad except and could never fail. The dual-stack test deploys an Instance and asserts both address families in host-route form, new tests cover a requested IPv4 address and the subnet's .0 address, and the operator-chosen routed id no longer collides.

Corrects stale statements (gateway handling, on-link mechanism, ConfigDrive gate, DNS delivery, cidr parameter), documents the two-pass security-group structure, host protection and script contract, turns the checklist into an implementation status, and lists the review findings consciously deferred.

security_group.py drops the packet-mark scheme. Source-side rules RETURN allowed traffic instead of marking it, egress rules end in RETURN and the egress chain drops the rest, and FORWARD holds two fixed hooks into BF-L3-IN and BF-L3-OUT plus one ACCEPT per bridge for fabric-bound traffic. Until add_network_rules has run the egress chain holds a single DROP. The per-bridge teardown on Instance destroy is replaced by a sweep in the periodic cleanup_rules for bridges that no longer exist. A bridge port without programmed rules is no longer dropped by a backstop, the same gap classic bridges have. modifybrdr.sh replaces the host-wide BRDR-INPUT chain with two per-bridge INPUT rules that drop everything from the bridge that is not ICMP(v6), plus the existing IPv6 rpfilter rule, all removed with the bridge. The golden files and the design doc are excluded from the RAT check. Claude-Session: https://claude.ai/code/session_01LkswKyuC2a58YCHFTEPnay

…tadata An Instance with more than one SSH keypair gets its keys as a single newline-joined string. The OpenStack meta_data.json builder passed that string through as one key, so "keys" held a single object and "public_keys" a single map entry whose value contained both keys with a newline in between. cloud-init treats every public_keys value as one key and never splits it, so only the first key worked at best. The name derived from the third whitespace-separated token could also end up containing a newline, and the replace("\\n", "") calls stripped a literal backslash-n rather than a newline and did nothing. The builder now splits the content on line breaks and emits one "keys" object and one "public_keys" entry per key. A key is named after its comment when it has one no earlier key used, otherwise "key" for a lone key and key0, key1, ... when there are several.

wido added component:kvm component:networking component:advanced-networking labels Jul 31, 2026

boring-cyborg Bot added component:api component:integration-test component:marvin component:security-group component:storage component:UI Python Warning... Python code Ahead! labels Jul 31, 2026

wido mentioned this pull request Jul 31, 2026

network: L3 routed networking without shared L2 domain #12210

Open

wido requested review from kiranchavala and weizhouapache July 31, 2026 12:38

DaanHoogland requested a review from Copilot August 3, 2026 04:54

Copilot started reviewing on behalf of DaanHoogland August 3, 2026 04:55 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

DaanHoogland added this to CloudStack Testing Aug 31, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 31, 2026

DaanHoogland moved this from Backlog to conflict/waiting in CloudStack Testing Aug 31, 2026

wido force-pushed the direct-routed-network branch from bf3d8a3 to 526934c Compare September 4, 2026 11:59

boring-cyborg Bot added component:orchestration component:secondary-storage component:ssvm component:virtual-router labels Sep 4, 2026

boring-cyborg Bot added the component:vpc label Sep 5, 2026

DaanHoogland reviewed Sep 7, 2026

View reviewed changes

Comment thread ui/src/views/network/CreateL3NetworkForm.vue Outdated

DaanHoogland reviewed Sep 7, 2026

View reviewed changes

Comment thread ui/src/views/network/CreateL3NetworkForm.vue Outdated

DaanHoogland reviewed Sep 7, 2026

View reviewed changes

Comment thread server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java Outdated

wido force-pushed the direct-routed-network branch from a911e9c to 655530c Compare September 7, 2026 14:02

boring-cyborg Bot added the component:ipv6 label Sep 9, 2026

wido force-pushed the direct-routed-network branch 2 times, most recently from 0cc8922 to 8a81e58 Compare September 9, 2026 20:35

wido force-pushed the direct-routed-network branch from 8a81e58 to ff5a419 Compare September 10, 2026 04:59

wido added 8 commits September 10, 2026 09:46

boring-cyborg Bot added the component:build label Sep 14, 2026

Navigation Menu

Uh oh!

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758
wido wants to merge 10 commits into
apache:mainfrom
wido:direct-routed-network

wido commented Jul 31, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

kiranchavala commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

DaanHoogland commented Sep 1, 2026

Uh oh!

wido commented Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wido commented Sep 7, 2026

Uh oh!

wido commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

wido commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Management server

ConfigDrive

KVM agent

Other changes this PR makes to CloudStack

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

kiranchavala commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

DaanHoogland commented Sep 1, 2026

Uh oh!

wido commented Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wido commented Sep 7, 2026

Uh oh!

wido commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wido commented Jul 31, 2026 •

edited

Loading

codecov Bot commented Jul 31, 2026 •

edited

Loading