Skip to content

README.md

Latest commit

 

History

History
368 lines (259 loc) · 8.32 KB

README.md

File metadata and controls

368 lines (259 loc) · 8.32 KB
Raw

Phase 3: Traefik VM HA With Keepalived

This lab captures the manually validated Phase 3 gateway HA setup as version-controlled templates.

Client
  -> VIP 192.168.56.100
  -> Traefik-1 / Traefik-2
  -> backend-1 / backend-2

This is a local learning lab for Traefik gateway HA, Keepalived, VRRP, and VIP failover. It is not production hardening.

Final Topology

Node IP Role Priority Gateway Header
proxy-1 192.168.56.11 MASTER 110 X-Gateway-Node: traefik-1
proxy-2 192.168.56.12 BACKUP 100 X-Gateway-Node: traefik-2
VIP 192.168.56.100 floating IP - -
host/backend 192.168.56.1 backend host - -

Backend upstreams:

backend-1: http://192.168.56.1:8080
backend-2: http://192.168.56.1:8083

The backend layer is provided by the Phase 2 lab:

make traefik-backend-ha-stack-up

Prerequisites

  • Two Ubuntu Server VMs on the same host-only network.
  • Backend host reachable from both VMs at 192.168.56.1.
  • Phase 2 backend HA lab running on the host.
  • Traefik v3.7.1 binary installed on both proxy VMs.
  • Keepalived installed on both proxy VMs.
  • VM interface name is enp0s8.

Adjust the templates if your VM IPs, interface, or backend host IP differ.

VM Network Setup

Recommended VirtualBox host-only network:

Subnet: 192.168.56.0/24
Host/backend: 192.168.56.1
proxy-1: 192.168.56.11
proxy-2: 192.168.56.12
VIP: 192.168.56.100
Interface: enp0s8

Check the interface name on each VM:

ip addr

Check backend reachability from each proxy VM:

curl -i http://192.168.56.1:8080/health
curl -i http://192.168.56.1:8083/health

Install Traefik

Install Traefik v3.7.1 on each VM:

curl -L https://github.com/traefik/traefik/releases/download/v3.7.1/traefik_v3.7.1_linux_amd64.tar.gz -o traefik.tar.gz
tar -xzf traefik.tar.gz
sudo install -m 0755 traefik /usr/local/bin/traefik
traefik version

Create the Traefik config directory:

sudo mkdir -p /etc/traefik

Copy files to proxy-1:

scp deploy/availability-lab/traefik-vm-keepalived/proxy-1/traefik.yml user@192.168.56.11:/tmp/traefik.yml
scp deploy/availability-lab/traefik-vm-keepalived/proxy-1/dynamic.yml user@192.168.56.11:/tmp/dynamic.yml
scp deploy/availability-lab/traefik-vm-keepalived/proxy-1/traefik.service user@192.168.56.11:/tmp/traefik.service
ssh user@192.168.56.11 'sudo mv /tmp/traefik.yml /etc/traefik/traefik.yml && sudo mv /tmp/dynamic.yml /etc/traefik/dynamic.yml && sudo mv /tmp/traefik.service /etc/systemd/system/traefik.service'

Copy files to proxy-2:

scp deploy/availability-lab/traefik-vm-keepalived/proxy-2/traefik.yml user@192.168.56.12:/tmp/traefik.yml
scp deploy/availability-lab/traefik-vm-keepalived/proxy-2/dynamic.yml user@192.168.56.12:/tmp/dynamic.yml
scp deploy/availability-lab/traefik-vm-keepalived/proxy-2/traefik.service user@192.168.56.12:/tmp/traefik.service
ssh user@192.168.56.12 'sudo mv /tmp/traefik.yml /etc/traefik/traefik.yml && sudo mv /tmp/dynamic.yml /etc/traefik/dynamic.yml && sudo mv /tmp/traefik.service /etc/systemd/system/traefik.service'

Enable and start Traefik on both VMs:

sudo systemctl daemon-reload
sudo systemctl enable --now traefik
sudo systemctl status traefik

Install Keepalived

Install Keepalived on both VMs:

sudo apt update
sudo apt install -y keepalived curl

Enable non-local bind:

echo 'net.ipv4.ip_nonlocal_bind=1' | sudo tee /etc/sysctl.d/99-otp-ha.conf
sudo sysctl --system
sysctl net.ipv4.ip_nonlocal_bind

Create the script user:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin keepalived_script || true

Create Keepalived config directory:

sudo mkdir -p /etc/keepalived

Copy files to proxy-1:

scp deploy/availability-lab/traefik-vm-keepalived/proxy-1/keepalived.conf user@192.168.56.11:/tmp/keepalived.conf
scp deploy/availability-lab/traefik-vm-keepalived/proxy-1/check-traefik.sh user@192.168.56.11:/tmp/check-traefik.sh
ssh user@192.168.56.11 'sudo mv /tmp/keepalived.conf /etc/keepalived/keepalived.conf && sudo mv /tmp/check-traefik.sh /etc/keepalived/check-traefik.sh && sudo chmod 755 /etc/keepalived/check-traefik.sh'

Copy files to proxy-2:

scp deploy/availability-lab/traefik-vm-keepalived/proxy-2/keepalived.conf user@192.168.56.12:/tmp/keepalived.conf
scp deploy/availability-lab/traefik-vm-keepalived/proxy-2/check-traefik.sh user@192.168.56.12:/tmp/check-traefik.sh
ssh user@192.168.56.12 'sudo mv /tmp/keepalived.conf /etc/keepalived/keepalived.conf && sudo mv /tmp/check-traefik.sh /etc/keepalived/check-traefik.sh && sudo chmod 755 /etc/keepalived/check-traefik.sh'

Start Keepalived:

sudo systemctl enable --now keepalived
sudo systemctl status keepalived

Restart both services after config changes:

sudo systemctl restart traefik
sudo systemctl restart keepalived

Validate Each Proxy Directly

From the host/backend machine:

curl -i http://192.168.56.11/health
curl -i http://192.168.56.12/health
curl -I http://192.168.56.11:8080/dashboard/
curl -I http://192.168.56.12:8080/dashboard/

Expected headers:

proxy-1: X-Gateway-Node: traefik-1
proxy-2: X-Gateway-Node: traefik-2

Traefik ping endpoint:

curl -i http://192.168.56.11:8080/ping
curl -i http://192.168.56.12:8080/ping

Validate VIP

On proxy-1:

ip addr show enp0s8

Expected when proxy-1 is MASTER:

192.168.56.100/24

From the host/backend machine:

curl -i http://192.168.56.100/health

Expected:

HTTP/1.1 200 OK
X-Gateway-Node: traefik-1

Failover Loop

Run this from the host/backend machine:

while true; do
  date +"%H:%M:%S"
  curl -s -i http://192.168.56.100/health | grep -E 'HTTP/|X-Gateway-Node'
  sleep 0.5
done

Traefik Failure Test

Stop Traefik on proxy-1:

ssh user@192.168.56.11 'sudo systemctl stop traefik'

Expected:

  • Keepalived on proxy-1 loses the Traefik health check.
  • VIP migrates to proxy-2.
  • The failover loop shows X-Gateway-Node: traefik-2.

Start Traefik again on proxy-1:

ssh user@192.168.56.11 'sudo systemctl start traefik'

Expected:

  • proxy-1 health recovers.
  • With the current priority-based config, VIP fails back to proxy-1.
  • The failover loop shows X-Gateway-Node: traefik-1.

Keepalived Failure Test

Stop Keepalived on proxy-1:

ssh user@192.168.56.11 'sudo systemctl stop keepalived'

Expected:

  • VIP migrates to proxy-2.
  • Traffic continues through proxy-2.

Restart Keepalived on proxy-1:

ssh user@192.168.56.11 'sudo systemctl start keepalived'

Reboot Validation

Reboot proxy-1:

ssh user@192.168.56.11 'sudo reboot'

Expected:

  • VIP migrates to proxy-2 while proxy-1 is down.
  • When proxy-1 returns and Traefik/Keepalived are healthy, VIP fails back.

Reboot both VMs one at a time and confirm:

curl -i http://192.168.56.100/health

still recovers after services start.

Troubleshooting

VIP ping works but curl VIP fails

Enable non-local bind and restart services:

echo 'net.ipv4.ip_nonlocal_bind=1' | sudo tee /etc/sysctl.d/99-otp-ha.conf
sudo sysctl --system
sudo systemctl restart traefik
sudo systemctl restart keepalived

VIP is not assigned

Check Keepalived and the VM interface:

sudo systemctl status keepalived
journalctl -u keepalived -n 100 --no-pager
ip addr show enp0s8

Both nodes become MASTER

Check:

  • both VMs are on the same VRRP-capable network
  • interface is enp0s8
  • both configs use virtual_router_id 51
  • both configs use auth_pass otp-ha
  • VirtualBox host-only network allows traffic between VMs
  • firewall is not blocking VRRP protocol 112

Traefik returns 502 or 503

Check backend-1/backend-2 on the host:

curl -i http://192.168.56.1:8080/health
curl -i http://192.168.56.1:8083/health
make traefik-backend-ha-ps

Traefik dashboard

The dashboard is enabled with insecure API mode for this local lab only:

http://192.168.56.11:8080/dashboard/
http://192.168.56.12:8080/dashboard/

Do not expose this configuration as-is in production.

Deferred

  • Nginx HA version
  • TLS / HTTPS
  • production dashboard security
  • Redis/PostgreSQL/Mongo HA
  • Kubernetes ingress HA
  • cloud load balancer integration
  • metrics/tracing for failover timing