Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to Instances

Add a guest network type in which the hypervisor performs L3 routing
for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance
receives a public IPv4 address as a /32 and/or an IPv6 address as a
/128, with a shared, host-independent link-local gateway (169.254.0.1
and fe80::1) that every hypervisor carries. All addressing reaches the
Instance exclusively via ConfigDrive/cloud-init; a routing daemon on
the host (FRR, BIRD, ...) advertises the addresses to the fabric and is
deliberately out of scope for CloudStack.

The networks live on a dedicated physical network with the new ROUTED
isolation method, so the feature can be added to existing zones without
touching anything already running there. Each network carries a
routed://<id> broadcast domain - the id allocated from the physical
network's vnet range or chosen by the operator - naming the network's
uplink-less bridge (brdr-<id>) that the new modifybrdr.sh manages on
every host: gateway addresses, forwarding sysctls, strict rp_filter,
and a MAC derived from the routed id so it is identical on every
hypervisor and live migration never invalidates the guest's neighbour
cache. The existing modifymacip.sh installs the per-address host route
and static neighbour entry when a NIC is plugged.

Both address families are optional and IPv6-only networks are
supported: IPv4 is a subnet (cidr=..., or netmask + start/end IP), IPv6
is an ip6cidr alone - addresses derive from the subnet and the NIC MAC
with EUI-64, so no range exists. No gateways are declared or stored;
createNetwork gains an optional cidr parameter (L3-only, additive).
Security groups work through a --directrouted dispatch in
security_group.py that matches the return path by ipset destination
instead of bridge port; a golden-file harness pins the classic rule
stream to prove existing deployments unchanged. SystemVMs run on routed
public ranges (createVlanIpRange with vlan=routed://<id>) in the same
host-route form. ConfigDrive network data is always generated for these
NICs, with the IPv4 default route emitted as a network-level gateway
key so cloud-init (netplan >= 23.1, networkd >= 24.2) renders it
on-link. A DefaultL3NetworkOffering (UserData and DNS via ConfigDrive,
security groups) is created on install and upgrade, and the UI gains an
L3 creation form, ROUTED in the zone wizard, and active IPv4/IPv6
addresses in the Instance list with click-to-copy.

General fixes that the feature surfaced but apply beyond it:
auto-allocated ids are exempt from the dynamic-vlan-range check in
createVlanAndPublicIpRange; a failed VM start or migration prepare now
unplugs the NICs it plugged; canUseForDeploy() counts the real IPv4
pool and no longer hides IPv4-less networks from the deploy wizard; the
zone-wide IPv6 overlap check keys on ip6_cidr rather than ip6_gateway.

The design document with the decision log lives in docs/design/.

This implements #12210

Claude-Session: https://claude.ai/code/session_01LkswKyuC2a58YCHFTEPnay

Author: wido

api/src/main/java/com/cloud/network/Network.java

@@ -43,7 +43,7 @@
 public interface Network extends ControlledEntity, StateObject<Network.State>, InternalIdentity, Identity, Serializable, Displayable {
 
     enum GuestType {
-        Shared, Isolated, L2;
+        Shared, Isolated, L2, L3;
 
         public static GuestType fromValue(String type) {
             if (StringUtils.isBlank(type)) {
@@ -54,6 +54,8 @@ public static GuestType fromValue(String type) {
                 return Isolated;
             } else if (type.equalsIgnoreCase("L2")) {
                 return L2;
+            } else if (type.equalsIgnoreCase("L3")) {
+                return L3;
             } else {
                 throw new InvalidParameterValueException("Unexpected Guest type : " + type);
             }

api/src/main/java/com/cloud/network/Networks.java

@@ -131,6 +131,23 @@ public <T> URI toUri(T value) {
                 }
             }
         },
+        /**
+         * Direct Routed (L3) networks: the id is a label naming the per-network bridge on the
+         * hypervisor (brdr-&lt;id&gt;), not an encapsulation — nothing appears on the wire.
+         */
+        Routed("routed", Long.class) {
+            @Override
+            public <T> URI toUri(T value) {
+                try {
+                    if (value.toString().contains("://"))
+                        return new URI(value.toString());
+                    else
+                        return new URI("routed://" + value.toString());
+                } catch (URISyntaxException e) {
+                    throw new CloudRuntimeException("Unable to convert to broadcast URI: " + value);
+                }
+            }
+        },
         UnDecided(null, null),
         OpenDaylight("opendaylight", String.class),
         TUNGSTEN("tf", String.class),

api/src/main/java/com/cloud/offering/NetworkOffering.java

@@ -81,6 +81,7 @@ enum RoutingMode {
     public final static String DefaultL2NetworkOfferingVlan = "DefaultL2NetworkOfferingVlan";
     public final static String DefaultL2NetworkOfferingConfigDrive = "DefaultL2NetworkOfferingConfigDrive";
     public final static String DefaultL2NetworkOfferingConfigDriveVlan = "DefaultL2NetworkOfferingConfigDriveVlan";
+    public final static String DefaultL3NetworkOffering = "DefaultL3NetworkOffering";
 
     /**
      * @return name for the network offering.

api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java

@@ -87,6 +87,12 @@ public class CreateNetworkCmd extends BaseCmd implements UserCmd {
         + "for shared networks and isolated networks when it belongs to VPC")
     private String netmask;
 
+    @Parameter(name = ApiConstants.CIDR, type = CommandType.STRING, since = "24.0.0",
+        description = "The IPv4 subnet of the Network in CIDR notation, e.g. 192.0.2.0/24. Supported for L3 (Direct Routed) "
+        + "networks only, as an alternative to netmask: the IP range defaults to the subnet's usable addresses, "
+        + "and startip/endip may narrow it")
+    private String cidr;
+
     @Parameter(name = ApiConstants.START_IP, type = CommandType.STRING, description = "The beginning IP address in the Network IP range")
     private String startIp;
 
@@ -231,6 +237,10 @@ public String getNetmask() {
         return netmask;
     }
 
+    public String getCidr() {
+        return cidr;
+    }
+
     public String getStartIp() {
         return startIp;
     }
@@ -340,10 +350,10 @@ public Long getPhysicalNetworkId() {
             }
         }
         if (physicalNetworkId != null) {
-            if ((offering.getGuestType() == GuestType.Shared) || (offering.getGuestType() == GuestType.L2)) {
+            if ((offering.getGuestType() == GuestType.Shared) || (offering.getGuestType() == GuestType.L2) || (offering.getGuestType() == GuestType.L3)) {
                 return physicalNetworkId;
             } else {
-                throw new InvalidParameterValueException("Physical network ID can be specified for networks of guest IP type " + GuestType.Shared + " or " + GuestType.L2 + " only.");
+                throw new InvalidParameterValueException(String.format("Physical network ID can be specified for networks of guest IP type %s, %s or %s only.", GuestType.Shared, GuestType.L2, GuestType.L3));
             }
         } else {
             if (zoneId == null) {

api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java

@@ -124,6 +124,16 @@ public NetworkType getNetworkType() {
     }
 
 
+    /**
+     * A Direct Routed (L3) network needs the agent told when a secondary IP goes away, so the
+     * host route and neighbour entry are removed - otherwise the host keeps routing an address
+     * the Instance no longer owns, and the routing daemon keeps advertising it.
+     */
+    private boolean isDirectRoutedNetwork() {
+        Network ntwk = _entityMgr.findById(Network.class, getNetworkId());
+        return ntwk != null && Network.GuestType.L3.equals(ntwk.getGuestType());
+    }
+
     private boolean isZoneSGEnabled() {
         Network ntwk = _entityMgr.findById(Network.class, getNetworkId());
         DataCenter dc = _entityMgr.findById(DataCenter.class, ntwk.getDataCenterId());
@@ -144,7 +154,7 @@ public void execute() throws InvalidParameterValueException {
             secIp = nicSecIp.getIp6Address();
         }
 
-        if (isZoneSGEnabled()) {
+        if (isZoneSGEnabled() || isDirectRoutedNetwork()) {
             //remove the security group rules for this secondary ip
             boolean success = false;
             success = _securityGroupService.securityGroupRulesForVmSecIp(nicSecIp.getNicId(), secIp, false);

api/src/test/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmdTest.java

@@ -251,7 +251,7 @@ public void testGetPhysicalNetworkIdForNonSharedNet() {
         try {
             cmd.getPhysicalNetworkId();
         } catch (Exception e) {
-            Assert.assertTrue(e.getMessage().startsWith("Physical network ID can be specified for networks of guest IP type Shared or L2 only."));
+            Assert.assertTrue(e.getMessage().startsWith("Physical network ID can be specified for networks of guest IP type Shared, L2 or L3 only."));
         }
     }
 

core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java

@@ -28,12 +28,28 @@ public class NetworkRulesVmSecondaryIpCommand extends Command {
     private String vmSecIp;
     private String vmMac;
     private String action;
+    private boolean directRouted;
+    private boolean applySecurityGroupRules = true;
 
     public NetworkRulesVmSecondaryIpCommand(String vmName, VirtualMachine.Type type) {
         this.vmName = vmName;
         this.type = type;
     }
 
+    public NetworkRulesVmSecondaryIpCommand(String vmName, String vmMac, String secondaryIp, boolean action, boolean directRouted, boolean applySecurityGroupRules) {
+        this(vmName, vmMac, secondaryIp, action);
+        this.directRouted = directRouted;
+        this.applySecurityGroupRules = applySecurityGroupRules;
+    }
+
+    public boolean isDirectRouted() {
+        return directRouted;
+    }
+
+    public boolean isApplySecurityGroupRules() {
+        return applySecurityGroupRules;
+    }
+
     public NetworkRulesVmSecondaryIpCommand(String vmName, String vmMac, String secondaryIp, boolean action) {
         this.vmName = vmName;
         this.vmMac = vmMac;