Skip to content

Add BYO CNI cluster automation to hack/aks/Makefile #3846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Add BYO CNI cluster automation to hack/aks/Makefile #3846

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3c52106
Initial plan
Copilot Jul 18, 2025
b771da8
Add BYO CNI cluster automation to hack/aks/Makefile
Copilot Jul 18, 2025
7a08a48
Add dual-stack support and comprehensive testing for BYO CNI automation
Copilot Jul 18, 2025
7269168
Add VM SKU parameter, Azure CNI support, and improved image registry …
Copilot Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions hack/aks/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,18 @@ CLUSTER ?= $(USER)-$(REGION)
GROUP ?= $(CLUSTER)
VNET ?= $(CLUSTER)

# BYO CNI cluster configuration
CNI_TYPE ?= cilium
CNS_VERSION ?= v1.5.38
AZURE_IPAM_VERSION ?= v0.3.0
CNS_IMAGE_REPO ?= MCR
CILIUM_DIR ?= 1.14
CILIUM_VERSION_TAG ?= v1.14.8
CILIUM_IMAGE_REGISTRY ?= acnpublic.azurecr.io
IPV6_HP_BPF_VERSION ?= v0.0.3
DUALSTACK ?= false
REPO_ROOT ?= $(shell git rev-parse --show-toplevel)

# Long Term Support (LTS)
ifeq ($(LTS),true)
LTS_ARGS=--k8s-support-plan AKSLongTermSupport --tier premium
Expand Down Expand Up @@ -109,6 +121,16 @@ vars: ## Show the input vars configured for the cluster commands
@echo K8S_VER=$(K8S_VER)
@echo LTS_ARGS=$(if $(LTS_ARGS),$(LTS_ARGS),$(LTS))
@echo COMMON_AKS_FIELDS=$(COMMON_AKS_FIELDS)
@echo CNI_TYPE=$(CNI_TYPE)
@echo CNS_VERSION=$(CNS_VERSION)
@echo AZURE_IPAM_VERSION=$(AZURE_IPAM_VERSION)
@echo CNS_IMAGE_REPO=$(CNS_IMAGE_REPO)
@echo CILIUM_DIR=$(CILIUM_DIR)
@echo CILIUM_VERSION_TAG=$(CILIUM_VERSION_TAG)
@echo CILIUM_IMAGE_REGISTRY=$(CILIUM_IMAGE_REGISTRY)
@echo IPV6_HP_BPF_VERSION=$(IPV6_HP_BPF_VERSION)
@echo DUALSTACK=$(DUALSTACK)
@echo REPO_ROOT=$(REPO_ROOT)


##@ SWIFT Infra
Expand Down Expand Up @@ -413,3 +435,110 @@ restart-vmss: ## Restarts the nodes in the cluster

scale-nodes: ## Scales the nodes in the cluster
$(AZCLI) aks nodepool scale --resource-group $(GROUP) --cluster-name $(CLUSTER) --name $(NODEPOOL) --node-count $(NODE_COUNT)

##@ BYO CNI Automation

byocni-cluster-up: ## Create complete BYO CNI cluster with CNS and CNI (default: Cilium)
@echo "Creating BYO CNI cluster with CNS and $(CNI_TYPE)..."
@echo "Variables: CLUSTER=$(CLUSTER), CNS_VERSION=$(CNS_VERSION), CNI_TYPE=$(CNI_TYPE)"
@$(MAKE) validate-cni-type
ifeq ($(CNI_TYPE),azurecni)
@$(MAKE) overlay-up
else
@$(MAKE) overlay-byocni-nokubeproxy-up
endif
@echo "Cluster created successfully. Deploying CNS..."
@$(MAKE) deploy-cns
@echo "CNS deployed successfully. Deploying $(CNI_TYPE)..."
ifeq ($(CNI_TYPE),cilium)
@$(MAKE) deploy-cilium
else ifeq ($(CNI_TYPE),azurecni)
@echo "Azure CNI is already configured in the cluster. No additional CNI deployment needed."
else
@echo "Warning: CNI_TYPE=$(CNI_TYPE) not supported yet."
@echo "Available CNI types: cilium, azurecni"
@exit 1
endif
@echo "BYO CNI cluster setup completed successfully!"

validate-cni-type: ## Validate the CNI type
ifeq ($(CNI_TYPE),cilium)
@echo "✓ CNI type validation passed: $(CNI_TYPE)"
else ifeq ($(CNI_TYPE),azurecni)
@echo "✓ CNI type validation passed: $(CNI_TYPE)"
else
@echo "✗ Error: CNI_TYPE=$(CNI_TYPE) is not supported."
@echo "Available CNI types: cilium, azurecni"
@echo "Example: make byocni-cluster-up CNI_TYPE=cilium"
@echo "Example: make byocni-cluster-up CNI_TYPE=azurecni"
@exit 1
endif

deploy-cns: ## Deploy CNS to the cluster
@echo "Deploying CNS with version $(CNS_VERSION)..."
cd $(REPO_ROOT) && sudo -E env "PATH=$$PATH" make test-load \
CNS_ONLY=true \
CNS_VERSION=$(CNS_VERSION) \
AZURE_IPAM_VERSION=$(AZURE_IPAM_VERSION) \
INSTALL_CNS=true \
INSTALL_OVERLAY=true \
CNS_IMAGE_REPO=$(CNS_IMAGE_REPO)

deploy-cilium: ## Deploy Cilium to the cluster
@echo "Deploying Cilium $(CILIUM_VERSION_TAG) from directory v$(CILIUM_DIR)..."
@if [ ! -d "$(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)" ]; then \
echo "Error: Cilium directory v$(CILIUM_DIR) not found."; \
echo "Available versions: $$(ls $(REPO_ROOT)/test/integration/manifests/cilium/ | grep '^v' | tr '\n' ' ')"; \
exit 1; \
fi
ifeq ($(DUALSTACK),true)
@echo "Deploying Cilium with dual-stack configuration..."
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-config/cilium-config-dualstack.yaml
else
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-config/cilium-config.yaml
endif
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/files
Copy link
Preview

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path 'cilium-operator/files' appears to be a directory being applied as a file. This should likely be 'cilium-operator/files/' with trailing slash or use '-f' with '--recursive' flag, or specify individual YAML files.

Suggested change
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/files
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/files/

Copilot uses AI. Check for mistakes.

kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-agent/files
Comment on lines +500 to +501
Copy link
Preview

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path 'cilium-agent/files' appears to be a directory being applied as a file. This should likely be 'cilium-agent/files/' with trailing slash or use '-f' with '--recursive' flag, or specify individual YAML files.

Suggested change
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/files
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-agent/files
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/files --recursive
kubectl apply -f $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-agent/files --recursive

Copilot uses AI. Check for mistakes.

@export CILIUM_VERSION_TAG=$(CILIUM_VERSION_TAG) && \
Copy link
Preview

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The complex multi-line export and envsubst command should be broken into separate variables or a shell script for better readability and maintainability. Consider defining the environment variables at the target level or using a helper script.

Copilot uses AI. Check for mistakes.

export CILIUM_IMAGE_REGISTRY=$(CILIUM_IMAGE_REGISTRY) && \
export IPV6_HP_BPF_VERSION=$(IPV6_HP_BPF_VERSION) && \
envsubst '$${CILIUM_VERSION_TAG},$${CILIUM_IMAGE_REGISTRY},$${IPV6_HP_BPF_VERSION}' < $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-operator/templates/deployment.yaml | kubectl apply -f - && \
envsubst '$${CILIUM_VERSION_TAG},$${CILIUM_IMAGE_REGISTRY},$${IPV6_HP_BPF_VERSION}' < $(REPO_ROOT)/test/integration/manifests/cilium/v$(CILIUM_DIR)/cilium-agent/templates/daemonset.yaml | kubectl apply -f -

byocni-cluster-vars: ## Show variables for BYO CNI cluster setup
@echo "=== BYO CNI Cluster Configuration ==="
@echo "Basic cluster settings:"
@echo " CLUSTER=$(CLUSTER)"
@echo " GROUP=$(GROUP)"
@echo " REGION=$(REGION)"
@echo " SUB=$(SUB)"
@echo " VNET=$(VNET)"
@echo " VM_SIZE=$(VM_SIZE)"
@echo ""
@echo "CNI configuration:"
@echo " CNI_TYPE=$(CNI_TYPE)"
@echo ""
@echo "CNS configuration:"
@echo " CNS_VERSION=$(CNS_VERSION)"
@echo " AZURE_IPAM_VERSION=$(AZURE_IPAM_VERSION)"
@echo " CNS_IMAGE_REPO=$(CNS_IMAGE_REPO) (MCR/ACR - affects CNS image paths)"
@echo ""
@echo "Cilium configuration:"
@echo " CILIUM_DIR=$(CILIUM_DIR)"
@echo " CILIUM_VERSION_TAG=$(CILIUM_VERSION_TAG)"
@echo " CILIUM_IMAGE_REGISTRY=$(CILIUM_IMAGE_REGISTRY)"
@echo " IPV6_HP_BPF_VERSION=$(IPV6_HP_BPF_VERSION)"
@echo " DUALSTACK=$(DUALSTACK)"
@echo ""
@echo "Image registry options:"
@echo " - MCR: mcr.microsoft.com/containernetworking"
@echo " - ACR: acnpublic.azurecr.io (default for Cilium)"
@echo " - Custom: your-registry.azurecr.io/path"
@echo ""
@echo "Repository root:"
@echo " REPO_ROOT=$(REPO_ROOT)"
@echo ""
@echo "Example usage:"
@echo " make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id>"
@echo " make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id> CNI_TYPE=azurecni"
@echo " make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id> CNS_VERSION=v1.6.0 CILIUM_DIR=1.16 CILIUM_VERSION_TAG=v1.16.5"
96 changes: 96 additions & 0 deletions hack/aks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,100 @@ AKS Clusters
windows-nodepool-up Add windows node pool
down Delete the cluster
vmss-restart Restart the nodes of the cluster

BYO CNI Automation
byocni-cluster-up Create complete BYO CNI cluster with CNS and CNI (default: Cilium)
deploy-cns Deploy CNS to the cluster
deploy-cilium Deploy Cilium to the cluster
byocni-cluster-vars Show variables for BYO CNI cluster setup
validate-cni-type Validate the CNI type
```

## BYO CNI Cluster Automation

The Makefile now includes automated setup for complete BYO CNI clusters with CNS and CNI deployment.

### Quick Start

Create a BYO CNI cluster with Cilium (default):
```bash
make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id>
```

Create a BYO CNI cluster with Azure CNI:
```bash
make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id> CNI_TYPE=azurecni
Comment on lines +68 to +70
Copy link
Preview

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example for Azure CNI deployment is misleading. Based on the Makefile logic, Azure CNI uses the 'overlay-up' target which creates a cluster with Azure CNI pre-configured, but the example suggests it's a BYO CNI setup like Cilium.

Suggested change
Create a BYO CNI cluster with Azure CNI:
```bash
make byocni-cluster-up CLUSTER=my-cluster SUB=<subscription-id> CNI_TYPE=azurecni
Create a cluster with Azure CNI pre-configured:
```bash
make overlay-up CLUSTER=my-cluster SUB=<subscription-id>

Copilot uses AI. Check for mistakes.

```

### Customization

All parameters are configurable:
```bash
make byocni-cluster-up \
CLUSTER=my-cilium-cluster \
SUB=<subscription-id> \
VM_SIZE=Standard_D2s_v3 \
CNS_VERSION=v1.6.0 \
CILIUM_DIR=1.16 \
CILIUM_VERSION_TAG=v1.16.5 \
CILIUM_IMAGE_REGISTRY=mcr.microsoft.com/containernetworking
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add node vm sku as parameter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added VM_SIZE as a configurable parameter for BYO CNI cluster automation. It's now documented in the configuration section and included in the customization examples. Commit 7269168

```

Using different image repositories:
```bash
# Using MCR for both CNS and Cilium
make byocni-cluster-up \
CLUSTER=my-cluster \
SUB=<subscription-id> \
CNS_IMAGE_REPO=MCR \
CILIUM_IMAGE_REGISTRY=mcr.microsoft.com/containernetworking

# Using ACR for CNS and custom registry for Cilium
make byocni-cluster-up \
CLUSTER=my-cluster \
SUB=<subscription-id> \
CNS_IMAGE_REPO=ACR \
CILIUM_IMAGE_REGISTRY=my-registry.azurecr.io/cilium
```

### Available Configuration

- `CNI_TYPE`: cilium, azurecni (default: cilium)
- `VM_SIZE`: Node VM size (default: Standard_B2s)
- `CNS_VERSION`: CNS version to deploy (default: v1.5.38)
- `AZURE_IPAM_VERSION`: Azure IPAM version (default: v0.3.0)
- `CNS_IMAGE_REPO`: CNS image repository - MCR or ACR (default: MCR)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on image repo, can cns/cilium image path be constructed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enhanced documentation to clarify the relationship between image repositories and image paths. Added examples showing different registry configurations (MCR vs ACR vs custom) and improved the variable display to show how CNS_IMAGE_REPO affects CNS image paths while CILIUM_IMAGE_REGISTRY controls Cilium images. Commit 7269168

- MCR: Uses Microsoft Container Registry paths
- ACR: Uses Azure Container Registry paths
- `CILIUM_DIR`: Cilium version directory - 1.12, 1.13, 1.14, 1.16, 1.17 (default: 1.14)
- `CILIUM_VERSION_TAG`: Cilium image tag (default: v1.14.8)
- `CILIUM_IMAGE_REGISTRY`: Cilium image registry (default: acnpublic.azurecr.io)
- Can be set to mcr.microsoft.com/containernetworking for MCR
- Or custom registry URL
- `IPV6_HP_BPF_VERSION`: IPv6 HP BPF version for dual stack (default: v0.0.3)
- `DUALSTACK`: Enable dual-stack configuration (default: false)

View all configuration variables:
```bash
make byocni-cluster-vars
```

### Workflow

The `byocni-cluster-up` target orchestrates the complete setup workflow:

**For Cilium CNI (default):**
1. **Cluster Creation**: Uses `overlay-byocni-nokubeproxy-up` to create AKS cluster without CNI
2. **CNS Deployment**: Uses root makefile `test-load` target with CNS-specific parameters
3. **CNI Deployment**: Deploys Cilium using manifests from `test/integration/manifests/cilium/`

**For Azure CNI:**
1. **Cluster Creation**: Uses `overlay-up` to create AKS cluster with Azure CNI pre-configured
2. **CNS Deployment**: Uses root makefile `test-load` target with CNS-specific parameters
3. **CNI Configuration**: Azure CNI is already configured - no additional deployment needed

Individual steps can also be run separately:
```bash
make deploy-cns
make deploy-cilium
```
Loading