Here I document everything about my networking lab which serves as a practical ground for learing modern datacenter technologies.
My main focus is building scalable L3-only network fabrics
Caution
There has been a big issue with the last network configuration especially on PVE. Contact with web GUI was completly cut off and even OSPF backbone went down.
All because of a massive broadcast storm and loops that occured between the CRS326 and the Proxmox VE.
Please read OSPF and L2 Loop troubleshooting as it may be really relevant in case some other things might have stopped working.
I now used the correct approach to do what I intended but without the risk of a gigantic broadcast storm.
(It turned out that everything was caused by a L1 Loop)
Here I put things that I think are the most interesting and worth reading.
These are projects, case studies and troubleshooting logs.
-
School Network fried by lightning, WiFi fixing, Cisco WLC - Very hot topic right now, since my school does not have WiFi available for now. They basically asked me to fix their wifi. I will write here about what things I uncover since it is actually far more complicated than just WiFi.
-
eBGP Implementation between two AS'es - Plus DHCP Nightmare
-
Super Important OSPF and L2 Loop troubleshooting - This is important as it makes some changes to the entire lab archtecture.
-
Finally OSPF Implementation ! - Area 0 between CCR2004 and CRS326
-
L3 hardware offload instead of router-on-a-stick - Super fast port-speed connection for wide bandwith between Virtual Machines !
-
IPv6 - For now there is not much here since it's hard to get an IPv4 from my ISP.
This repository is structured to be a clear and useful reference. Here’s a map of the key directories:
/projects/: Probably the most interesting directory cause it's where all project documentations are.- /[device-name]/ (e.g.,
./ccr2004/,./r710/): Contains the latest configuration files and documentation for each piece of hardware. This is the source of truth for device settings. /IaC/: Holds all Infrastructure as Code projects, primarily using Terraform to automate deployments on Proxmox./docs/: Contains details about plans for improving the lab. For example a better addressation plan
Here is the simplified diagram which shows what Im making my network to look like.
Below is a really in-depth and specific explaination of my lab. However, if you do not want to read everything (though I encourage you to, since I think it's pretty interesting), I can short it for you.
Basically, there are:
- two routers connected with eBGP with failover,
- proxmox server
- VyOS, vSRX3 NGFWs/routers on Proxmox VE
- 10GbE fiber links
- Plans for BGP EVPN implementation (I will get into this after passing CCNA)
However, as I said, I think you might actually enjoy reading this a lot longer description:
My network is oriented towards a datacenter-styled approach because that is the field that I would love to work in.
I am actively improving things to make this lab as much as possible like a real data center Spine-Leaf Design.
For now the Spine of my network consists of three routers, a MikroTik CCR2004-1G-12S+2XS, a CRS326-24S+2Q+RM and a VyOS vRouter. Together, those two physical routers are connected with two eBGP sessions through two 10GbE Fiber links. I wanted to enable ECMP between them, however, in RouterOS 7.19.4, ECMP for BGP is not supported.
The CCR2004 has an ASN of 65000, the CRS326 has 65001 and the VyOS vRouter has a 65101 ASN.
The CCR2004 advertises the default route (0.0.0.0/0) to the CRS326 which advertises Networks from it's BGP_ADV_NET address list.
There is also a DHCP Server running on a loopback-like interface on the CCR2004. This ensures that it is reachable even when one of the links go down and removes the need for two identical DHCP Servers for two different interfaces, since it's listening on a single bridge.
From the outside, this network might look pretty small. However, a lot happens in Proxmox Virtual Environment which runs on my Dell PowerEdge R710.
In the PVE I run a lot of networking appliances like VyOS and vSRX3 NGFWs/Routers.
The PVE is connected through a single DAC 10GbE cable to the CRS326.
This single cable carries a lot of tagged traffic which then gets switched by the main vmbr0 bridge.
This way, the inter-VLAN routing between VMs and for example Kubernetes Cluster gets handled by the CRS326 which has enabled L3 Hardware offloading.
Even though the L3HW offload on CRS326 is fairly simple, as anything above simple routing gets handed to the CPU, it allows for line-speed (10GbE) routing between different VLANs which live on the PVE server.
That is of course a bit of hairpinning since the traffic goes twice through the same physical cable.
VLAN segmentation on that link between CRS326 and PVE, allows me to create different logical links without worrying about buying another SFP+ NICs.
USERS_NET is currently reachable through the point-to-point link between the CRS326 and a VyOS vRouter which also travels through that same DAC Cable but is of course separated with VLAN tagging.
There is also eBGP session running on the point-to-point link between the CRS326 and the VyOS router.
USERS_NET Access is available for PCs in another room next to mine via a dual-port RJ45 10GbE NIC which has both its ports bridged onto vmbr-users, where also the mentioned VyOS Router has one of it's interfaces.
This VyOS vRouter is also a DHCP Relay for all devices in USERS_NET.
For now, for management and access to the entire network I used a kind of a master-network which was available through the ether1 interface on the CCR2004 router.
This is of course a bad practise and I will be deprecating it.
The reason why I got into thinking about this OOB-Only network is because I started learning more about real datacenter networking devices.
My MikroTik routers, even though powerful for their price, are not a real datacenter grade networking appliances.
However, my new Dell EMC S4048-ON is a real datacenter-grade device.
The mgmt interface on it is not just an another ethernet port with a mgmt text next to it.
Since this is a real datacenter switch, it features a physical separation of data and control plane, which means that this management port is physically connected differently than every other interface.
It's connected directly to the CPU, while every single other port is connected to the Trident 2 ASIC.
The same thing applies to the vSRX3 vRouters.
They also feature a control and data plane separation.
New approach will be to make networking devices management available only from a Out-of-band network.
Each device will get assigned an IP from the 10.9.0.0/24 network on a physical management interface. NOT on an SVI.
Accessing OOB network from anywhere other than a physical connection to the OOB switch will be impossible and vice-versa.
Proxmox Management will still be available from in-band network.
However, one problem with this occured when I thought about the vSRX3 vRouters in the PVE. How to make their fxp0.0 management interfaces available from the OOB Network?
The solution to this is to first create a new bridge in the PVE, for example vmbr-oob.
Then to make it physically separated, it's neccessary to add a physical interface to the vmbr-oob, for example eno1.
Then what's left is to connect the eno1 interface to the OOB switch and add all net0 vNICs for the vSRX3 vRouters to the vmbr-oob.
The first networking device (net0) added to the vSRX3 vRouter is always assigned as fxp0.0 port, which is specifically a management interface.
This way, the management interfaces for the vSRX3 appliances will be always reachable through the OOB network even when the main PVE link goes down.
This also improves security a lot.
I want to use this switch as the Spine switch in my lab.
It's absolutely insane and it's a gigantic upgrade for me.
After all, it's my first real datacenter-grade L3 fabric switch.
It will be a great L3 fabric for my network which will allow me to create super-fast links to my vSRX3 NGFWs/vMX Routers and handle VXLAN Tunnels.
I want to make my network more to be like a real datacenter network.
Now, hyperscalers use a L3-Only Spine-Leaf architecture in their datacenters.
A L3-Only network means that there is no stretching of L2 domains, no issues caused by LACP, no locked ports cause of STP and no fake load-balancing with bond interfaces.
By making my network L3-Only, I will be able to do everything a lot more elastically, and also utilize real load-balancing using ECMP for BGP.
Creating a stable L3-Only network fabric also makes up a great environment for a very advanced datacenter technologies like VXLAN which is currently used routinely in datacenters.
I would create a VXLAN tunnel between VMs_NET, which would be behind a vMX router, and USERS_NET, which would be reachable through a vSRX3 NGFW.
The vMX and vSRX would be the VTEPs for this tunnel and would maintain EVPN session and advertise l2vpn afi to the rest of the network.
Basically, there will be absolutely no L2 domain stretching, no STP, no LAG etc.
Typically there are VLANs connected to a switch and traffic is carried from a VLAN, to the L3 switch (or router, wherever the SVI/IRB is) and sent back to another VLAN.
I will be leaving that behind and everything will be routable.
For example:
USERS_NET - reachable through AS65103, vSRX3 NGFW, leaf-vsrx3-0.
VMS_NET - reachable through AS65102, vMX router, leaf-vmx-0.
etc..
This is just so much more elegant than messing with LACP and STP.
That is just what is done in datacenters.
I could then create a new network VMS_NET_1 and connect it to a new vMX router.
the new vMX router, leaf-vmx-1 for example, would get assigned a ASN of 65104 and would be connected to the Dell EMC S4048-ON Spine switch.
As you can see, this topology matches (in a miniature scale) the policy of a L3 Clos fabric in datacenters:
No leaf is connected to another leaf.
Then I could create an Anycast Gateway on the leaf-vmx-0 and leaf-vmx-1 on the side of their networks (VMS_NET & VMS_NET_1) on an IRB.
After this I would create BGP EVPN sessions by making the vMX leafs advertise a new afi, which is l2vpn.
This way, the VMs in VMS_NET and VMS_NET_1 could talk to each other like they were in the same L2 domain, but of course they are not.
The insanely powerful Trident 2 ASIC allows this Dell switch to perform VXLAN encapsulation, BGP EVPN etc. with incredible line-speed bandwidth.
However, in the scenario depicted above, EVPN would not be enabled on the Dell S4048-ON.
So I would simply add a new SVI which would be the Anycast gateway for a new network USERS_NET_1 and I would simply enable EVPN on this Dell and also on the leaf-vsrx3-0.
That is how I could access the USERS_NET, through a SFP RJ45 transceiver plugged into the Dell, like I would be in the same L2 domain.
This is what I want to achieve for now but first just need to finish CCNA.
A list of the key components in my lab. Click a device name to see its configuration files.
| Device Type | Model | Role in the Lab |
|---|---|---|
| Server Rack | HPE 10636 G2 | Central mounting point for all equipment. |
| PVE Server | Dell PowerEdge R710 | Main virtualization host, running Proxmox VE. |
| Border Leaf Router | MikroTik CCR2004 | Border leaf Router, provides access to the internet, NAT, DHCP Server on loopback interface, VPNs and main firewall for North-South traffic |
| Leaf Router | MikroTik CRS326 | Leaf router. For now handles BGP, inter-VLAN routing with line-speed thanks to L3HW offload |
| Spine Switch | Dell S4048-ON | For now, unused. However, after I finish CCNA, it will become the Spine switch for my network and will handle BGP EVPN, VXLANs and West-East traffic with its astronomic Trident 2 ASIC which TCAM memory I am learning about. |
| OOB Switch | Brocade FastIron LS648 | Switch for OOB Management. Handles single L2 domain. |
| 0U PDU | HP S1132 | Enterprise-grade Power Distribution Unit. |
