Shared network issue on advanced zone with security group

[TadiosAbebe]

Hello community,
This past few days i was testing shared network in advanced zone with security group and i think i am just missing a lot of understanding about it. and here is my issue

• created a shared network with custom vlan(12) which is out of range of specified for guest network(2048-3072)
• then as soon as i launch an instance on that shared network, other isolated networks connectivity stops working for the physical host where the shared network instance reside in.
  • so for example if i have a VR1 and Instance1 in physical host 5, and as soon as the shared network instance gets allocated on physical host 5, the connectivity between Instance1 -> VR1, VR1 -> internet stops working and the same for all others VR and instances on that host
• and I think i boiled down the above issue as being an iptable rules and here is my findings
  • iptables -L -n -v listing before creating a shared network and an instance on the shared network

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain FORWARD (policy DROP 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 DOCKER-USER  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
      0     0 DOCKER-ISOLATION-STAGE-1  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
      0     0 DOCKER     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
  
  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain DOCKER (1 references)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain DOCKER-ISOLATION-STAGE-1 (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain DOCKER-ISOLATION-STAGE-2 (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 DROP       0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain DOCKER-USER (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0
  

  • iptables -L -n -v listing after creating a shared network and an instance on the shared network

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain FORWARD (policy DROP 150 packets, 6817 bytes)
  pkts bytes target     prot opt in     out     source               destination         
  127  7160 BF-brbond0-12  0    --  *      brbond0-12  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-is-bridged
      0     0 BF-brbond0-12  0    --  brbond0-12 *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-is-bridged
      0     0 DROP       0    --  *      brbond0-12  0.0.0.0/0            0.0.0.0/0           
      0     0 DROP       0    --  brbond0-12 *       0.0.0.0/0            0.0.0.0/0           
  150  6817 DOCKER-USER  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  150  6817 DOCKER-ISOLATION-STAGE-1  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
      0     0 DOCKER     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 ACCEPT     0    --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
  
  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain BF-brbond0-12 (2 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
  127  7160 BF-brbond0-12-IN  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-is-in --physdev-is-bridged
  121  5168 BF-brbond0-12-OUT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-is-out --physdev-is-bridged
      0     0 ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out bond0.12 --physdev-is-bridged
  
  Chain BF-brbond0-12-IN (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      6  1992 i-167-1541-def  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged
  
  Chain BF-brbond0-12-OUT (1 references)
  pkts bytes target     prot opt in     out     source               destination         
  121  5168 i-167-1541-def  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
  
  Chain DOCKER (1 references)
  pkts bytes target     prot opt in     out     source               destination         
  
  Chain DOCKER-ISOLATION-STAGE-1 (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
  150  6817 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain DOCKER-ISOLATION-STAGE-2 (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 DROP       0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
      0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain DOCKER-USER (1 references)
  pkts bytes target     prot opt in     out     source               destination         
  150  6817 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain i-167-1541-VM (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
      0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain i-167-1541-VM-eg (1 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
      0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  
  Chain i-167-1541-def (2 references)
  pkts bytes target     prot opt in     out     source               destination         
      0     0 ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
      6  1992 ACCEPT     17   --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:68 dpt:67
      0     0 ACCEPT     17   --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet8 --physdev-is-bridged udp spt:67 dpt:68
      0     0 DROP       17   --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:67
      0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged ! match-set i-167-1541-VM src
  121  5168 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet8 --physdev-is-bridged ! match-set i-167-1541-VM dst
      0     0 RETURN     17   --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src udp dpt:53
      0     0 RETURN     6    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src tcp dpt:53
      0     0 i-167-1541-VM-eg  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src
      0     0 i-167-1541-VM  0    --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
  

As seen in the above output, the dropped packet in Chain FORWARD (policy DROP 150 packets, 6817 bytes) keeps just increasing by the seconds afterward. and I don't really understand why isolate network packets are getting dropped once i create a shared network.

• I think the DOCKER related chains in the iptables output are coming from cephadm since i am using compute and storage on the same nodes

I am running apache cloudstack 4.20.1 with ubuntu server 24.04. I would be glad if anyone can shade some light to this.

[weizhouapache]

just curious, how to create isolated network in advanced zone with security groups ?

[weizhouapache]

I actually enabled security group after the creating of the zone, so when initially launching the zone, it was just advanced zone, the i was able to create isolated network normally as in the docs. then i enabled the security group provider on the physical network when i intend to created shared networks. The isolated networks works just fine until there is an instance on a shared network where they reside on the same physical host. And once the shared network instance is allocated on a host all other network VR->Instance or VR ->internet stops working and only the instance on the shared network works.

@weizhouapache Not sure if that answers your question?

got it, so you are actually using advanced zone , and (1) isolated networks and (2) shared network with security group
right ?

[TadiosAbebe]

actually, here is what i did, there were a bunch of isolated networks and vm, i have 5 nodes in my zone all on the same cluster. so one day when i am testing the shared network scenario, all the vms and isolated network VRs stops working, i first thought i created a vlan conflict where i affected the other resources, but when i dig down i saw that the only affected vms and isolated network VRs are the one that resides on the same physical host as the one that have the vm on the shared network. then i migrated all isolated network VRs and instances from that host. So the above iptable rules are from one of the host where there is 1 isolated network VR, 1 instance on that isolated network and 1 instance on a shared network with security group(without VR)

[weizhouapache]

actually, here is what i did, there were a bunch of isolated networks and vm, i have 5 nodes in my zone all on the same cluster. so one day when i am testing the shared network scenario, all the vms and isolated network VRs stops working, i first thought i created a vlan conflict where i affected the other resources, but when i dig down i saw that the only affected vms and isolated network VRs are the one that resides on the same physical host as the one that have the vm on the shared network. then i migrated all isolated network VRs and instances from that host. So the above iptable rules are from one of the host where there is 1 isolated network VR, 1 instance on that isolated network and 1 instance on a shared network with security group(without VR)

Try out with removing the DOCKER* chains

[TadiosAbebe]

removed the docker* chains with the following command, but VR->internet and VM->VR connectivity issue still persists

iptables -D FORWARD -j DOCKER-USER
iptables -D FORWARD -j DOCKER-ISOLATION-STAGE-1
iptables -F DOCKER
iptables -F DOCKER-USER
iptables -F DOCKER-ISOLATION-STAGE-1
iptables -F DOCKER-ISOLATION-STAGE-2
iptables -X DOCKER
iptables -X DOCKER-USER
iptables -X DOCKER-ISOLATION-STAGE-1
iptables -X DOCKER-ISOLATION-STAGE-2

[TadiosAbebe]

The thing is, I couldn't not say the docker chain is dropping the traffic because that chain is on all the other hosts and it is not affecting them

One thing i know for sure right now is that, the security group iptables rules are interfering with the isolated network traffic. cause i created a new service offering without a security group and launching a vm on that security group doesn't interrupt existing isolated network traffic on the same host.

On a similar not i have a follow up question.

1. the reason i needed to create a shared network is mainly for public traffic for guest vms that need direct public ip to virtual machine without a VR, and i am only able to create the shared network on physical network 3(which is the bridge that have guest traffic type), will i be having any issue architecturally residing public traffic on guest traffic type

2. what issue would i be having if i create different shared network with the same vid(by enabling vlan id overlap) for different domains and set different start and end ip range. So for example

   • domain1 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.5 end_ip: 172.19.1.10]
   • domain2 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.11 end_ip: 172.19.1.15]
   • domain3 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.16 end_ip: 172.19.1.20]

• from my understanding if i do the above, user on domain1 can only use ip 172.19.1.5-172.19.1.10 securiy groups will prevent him from stealing and manually assigning an ip from domain2. is this correct?

[TadiosAbebe]

I was able to do some more test today spinning up a test Cloudstack environment and here are my findings,

• For a KVM host that doesn't have docker installed, the default FORWARD chain on iptables have an ACCEPT policy, and both instances on isolated network and instances on shared network with security group seems to have connectivity to the outside.
• As soon as docker is installed on that KVM host the default policy for the FORWARD Chain gets changed to DROP and instances on the isolated network loss connectivity to the outside( VM -> VR and VR to internet) while the instance that reside on the shared network with security group works fine.
• I guess the theory here is since shared network with securty groups add its own chain that allow or drop traffic the instance on the shared network keeps working without an issue
• And since isolated networks don't have specific rules the traffic gets caught by the docker DROP policy.
• Adding iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT will make instances on the isolated
  network to work again.

It would be great if some one with a better understanding of Linux networking shade some light on the above because i am not sure if this is exactly what is happening it is just my guess. Specifically i don't understand why the docker chain drop the traffic only when a shared network with security group is created on the host.

Anyways for my solution, instead of adding iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT, When I review the Cloudstack installation guide and I saw steps on setting the DEFAULT_FORWARD_POLICY to ACCEPT and enabling ufw, when i check my ufw it was inactive(don't remember why i have disabled it), but I add ufw rules and re-enabled ufw. Not sure if what resolved the connectivity issue is the DEFAULT_FORWARD_POLICY or other ufw rules, but it resolved the issue, now both the instances on the shared network and isolated network can communicate out to the internet.

Now getting back to my followup questions

1. the reason i needed to create a shared network is mainly for public traffic for guest vms that need direct public ip to virtual machine without a VR, and i am only able to create the shared network on physical network 3(which is the bridge that have guest traffic type), will i be having any issue architecturally residing public traffic on guest traffic type?

2. what issue would i be having if i create different shared network with the same vid(by enabling vlan id overlap) for different domains and set different start and end ip range. So for example

• domain1 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.5 end_ip: 172.19.1.10]
• domain2 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.11 end_ip: 172.19.1.15]
• domain3 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.16 end_ip: 172.19.1.20]

from my understanding if i do the above, user on domain1 can only use ip 172.19.1.5-172.19.1.10 securiy groups will prevent him from stealing and manually assigning an ip from domain2. is this correct?