DNS not working, even with super-simple setup

[sdlarsen]

Hi, trying to setup a cluster with terraform-hcloud-kubernetes, and the basics "just work". Control plane is up, my worker is up and all lights are green (v1.14.3) from talosctl health.
But when I try to install flux, the pre check always gives me:

kl -n flux-system flux2-flux-check-bl68l
► checking prerequisites
✗ Kubernetes API call failed: Get "https://10.21.0.1:443/version": dial tcp 10.21.0.1:443: i/o timeout

I tried adding a CiliumNetworkPolicy to no avail:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-flux-to-apiserver
  namespace: flux-system
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            kubernetes.io/service-name: kubernetes
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

I'm unsure where to look for clues, so suggestions welcome. My current configuration, stripped down to this:

module "kubernetes" {
  source  = "hcloud-k8s/kubernetes/hcloud"
  version = "3.14.3"

  hcloud_token = var.hcloud_token
  cluster_name = var.cluster_name

  # Node pools
  control_plane_nodepools      = 3 x cx23
  worker_nodepools             = 1 tainted cs33, 1 untained cx23
  cluster_autoscaler_nodepools = not used

  # Disable built-in features
  cert_manager_enabled   = false
  ingress_nginx_enabled  = false
  metrics_server_enabled = false

  # Enable control plane load-balancer
  kube_api_load_balancer_enabled = true

  # And use specified CIDRs
  network_ipv4_cidr = "10.100.0.0/16"

  # Output configuration file paths to environment folder
  cluster_kubeconfig_path  = "environments/${var.environment_name}/kubeconfig"
  cluster_talosconfig_path = "environments/${var.environment_name}/talosconfig"

  # Disable work scheduled on control plane nodes
  cluster_allow_scheduling_on_control_planes = false

  # Configure DNS name for api server
  kube_api_hostname = "api.${var.cluster_public_hostname}"

  # When we destroy, we destroy
  cluster_delete_protection = false
  cluster_graceful_destroy  = false

  # Enable cluster access from everywhere
  # (this is not dangerous, as certificates are required for access)
  firewall_api_source = ["0.0.0.0/0", "::/0"]
}

From my tainted worker:

kubectl run -n flux-system test -it --image=busybox --rm --overrides='{
  "spec": {
    "tolerations": [
      {
        "key": "midasenergy.com/role",
        "operator": "Equal",
        "value": "infra",
        "effect": "NoSchedule"
      }
    ],
    "securityContext": {
      "runAsNonRoot": true,
      "runAsUser": 1000,
      "seccompProfile": { "type": "RuntimeDefault" }
    },
    "containers": [{
      "name": "test",
      "image": "busybox",
      "securityContext": {
        "allowPrivilegeEscalation": false,
        "capabilities": { "drop": ["ALL"] }
      },
      "stdin": true,
      "tty": true
    }]
  }
}' -- sh
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
~ $ cat /etc/resolv.conf
search flux-system.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.100.96.10
options ndots:5
~ $ nslookup kubernetes.default.svc
Server:		10.100.96.10
Address:	10.100.96.10:53

** server can't find kubernetes.default.svc: NXDOMAIN

** server can't find kubernetes.default.svc: NXDOMAIN

~ $ nc -v -z 10.100.96.10 53
10.100.96.10 (10.100.96.10:53) open

So I can access dns, but not lookup kubernetes.default.svc. Looking up external IP's work (like google.com)

I'm puzzled :)

[M4t7e]

Hey @sdlarsen, it seems you accidentally closed this when you updated your post. I just reopened it.

Here are some hints that should lead you in the right direction:

• 10.21.0.1 does not belong to your cluster (10.100.0.0/16). You’re probably not using the correct kubeconfig.

• A NetworkPolicy isn’t required unless you’ve already added other egress rules

• If you set kube_api_hostname, it will be used for external access. You have to make sure it’s resolvable from your machine. See:

  Specifies the hostname for external access to the Kubernetes API server. This must be a valid domain name, set to the API's public IP address.

• BusyBox has some very poor binaries with reduced functionality. As you can see from its response, it literally tried kubernetes.default.svc while ignoring the search domains. Just use a different image like debian with proper tools and it will work. See:

  • Nslookup does not work in Kubernetes if omitting .svc.cluster.local suffix docker-library/busybox#61
  • Nslookup does not work in latest busybox image docker-library/busybox#48

I’d recommend starting with a small, minimal configuration without advanced settings. If you follow all steps from the installation section, you can directly use flux bootstrap ... to initialize Flux.

[sdlarsen]

Hi @M4t7e ,
Thank you for chipping in. The 10.100.0.0/16 is the hetzner network, 10.20.0.0 and 10.21.0.0 are CNI network cidrs for pods and services (IIRC they are 10.96.0.0 and something else in the default setup). That should be just fine. But I'll try without specifying them.

I did provide a small version from the getting started, and it works using dnsutils . The issue is not related to flux, it was just the first thing I wanted to setup.

The dns name is resolvable, that is not the issue.

I'll work my way through and keep you posted. Again, thank you for replying.

[sdlarsen]

Ah, I see Hetzner har L3 subnet isolation. You're right about those CIDRs :)

[sdlarsen]

Thanks a bunch. I have a working setup now