🐛 BUG: no connectivity between nebula nodes when using different CAs

[timteka]

What version of nebula are you using? (nebula -version)

1.9.7

What operating system are you using?

Linux

Describe the Bug

We've decided to split our staff into two teams. Team1 and Team2. Issued additional CA (say ca-2025-team2.crt). Also issued another CA as a replacement of old one and concatenated all of them into one ca.crt ( ca-2024.crt + ca-2025.crt + ca-2025-team2.crt). And.... got connectivity problems between various nodes. Lighthouse successfully handshakes all of them, but hosts can't ping each other, or sometimes only few ping replies are successful, then stuck. Ok, we resign team2 certs by ca-2025.crt and voila - all is good again. What can be the course of our problem?

Worth mentioning, nodes of Team2 can ping each other. Moreover, some of them can ping nodes of Team1. But almost none of them can ping the most distant nodes - our office. And nodes of Team2 are located in aws tokyo.

[JackDoan]

Hi @timteka!

Can you post the contents of the CAs (nebula-cert -print -path your-ca.crt), some certificates signed with each of them, and the config files of the hosts that failed to communicate? (please redact your private keys!)

One thing that can cause this behavior is if the addresses you assign to nodes don't all have a subnet in common. As a quick example:

• 10.2.0.1/24 and 10.3.0.1/24 CANNOT communicate
• 10.2.0.1/8 and 10.3.0.1/8 CAN communicate

Is there any chance you have address conflicts between the Nebula overlay and your various underlay networks?

Please forgive me if these questions sound "obvious", but you know much more about your network that I do, and it's always good to check the basics first. Feel free to ping me in #support in our slack instance too, if you think discussing there would be quicker/easier for you.

[timteka]

Dear Jack,
Everything was working properly before... 1 or 2 days. And the problem is only with aws instances. Maybe the punch-holing or sth stopped functioning as before. Our last changes were those games with CAs. But seems like it's not.
We even made firewall holes for incoming 4242. Maybe it's also not correct. Do ordinary nodes need to have any incoming ports specifically opened? Or only the lighthouses, sitting on white ips, and relays? And what if ee opened some incoming ports like 4242, could we've broken "normal" way of things?

[timteka]

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/l4.crt
  key: /etc/nebula/l4.key

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "10.145.0.1"
    - "10.145.0.2"

  local_allow_list:
    interfaces:
      'br0': true
    "10.10.1.16/32": true

relay:
  am_relay: false
  use_relays: false

static_host_map:
  "10.145.0.1": ["wan_ip1:42420"]
  "10.145.0.2": ["wan_ip2:42420"]

listen:
  host: 0.0.0.0
  port: 42420

punchy:
  punch: true
  respond: true

tun:
  dev: nebula0
  unsafe_routes:
    - route: 10.146.0.0/25  # ovpn admins via lighthousey
      via: 10.145.0.1
    - route: 10.146.1.0/25  # ovpn quants via lighthousey
      via: 10.145.0.1
    - route: 10.146.0.128/25  # ovpn admins via s2
      via: 10.145.0.14

logging:
  level: debug
  format: text

stats:
  type: prometheus
  listen: 0.0.0.0:42421
  path: /metrics
  namespace: prometheusns
  subsystem: nebula
  interval: 10s

firewall:
  outbound_action: drop
  inbound_action: drop

  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    # Default permissive inbound for team1
    - port: any
      proto: any
      groups:
        - team1
    # Default permissive inbound for team2
    - port: any
      proto: any
      groups:
        - team2

[timteka]

Seems like we had "the One direction" connectivity problems. Successful handshakes with the lighthouse from both nodes, but no traffic between them afterwards. The relaying helped us.

[johnmaguire]

Hi @timteka - are you still experiencing issues or did relay solve you problem? Some hosts behind certain NATs are unable to communicate without relays. If one side of the handshake has a port forwarded / firewall opened or is behind certain Nebula-friendly NATs, you should be able to avoid relays. If this isn't happening, please verify that AWS or iptables/ufw/nft aren't blocking traffic to that port.

If you are still having issues can you provide logs from the affected hosts during attempted handshakes?

[timteka]

If you are still having issues can you provide logs from the affected hosts during attempted handshakes?

Dear John, we're getting some strange stuff. E.g. in one cloud (say AliBaba/AWS) in the same vpc, same security groups and all, several instances operate properly via nebula, few - not.

We haven't switched all instances to relaying, we do it gradually. I'll try to gather some logs.

[JackDoan]

@timteka I'm not sure if this is relevant to your problem, but there is a typo in your config:

  addvertise_addr:
    - "<your-ip-here>:42420"

should be:

  advertise_addrs:
    - "<your-ip-here>:42420"

https://nebula.defined.net/docs/config/lighthouse/#lighthouseadvertise_addrs

[timteka]

Hi @timteka - are you still experiencing issues or did relay solve you problem?
If you are still having issues can you provide logs from the affected hosts during attempted handshakes?

Dear John, here they are. Problems with t2-t2 server:

time="2025-12-23T11:23:27+03:00" level=debug msg="Generated index" index=392915495
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.145.0.41 localIndexId:392915495] mapTotalSize:34 vpnIp:10.145.0.41]"
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message sent" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:2 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=392915495 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=debug msg="Generated index" index=823659089
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=debug msg="Generated index" index=2607192624
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=debug msg="Generated index" index=1633951754
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:27+03:00" level=debug msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2681881324 localIndex=2681881324 remoteIndex=0 udpAddrs="[35.73.119.11:42420 172.31.42.166:42420]" vpnIp=10.145.0.41
time="2025-12-23T11:23:28+03:00" level=debug msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2681881324 localIndex=2681881324 remoteIndex=0 udpAddrs="[35.73.119.11:42420 172.31.42.166:42420]" vpnIp=10.145.0.41
time="2025-12-23T11:23:29+03:00" level=debug msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2681881324 localIndex=2681881324 remoteIndex=0 udpAddrs="[35.73.119.11:42420 172.31.42.166:42420]" vpnIp=10.145.0.41
time="2025-12-23T11:23:30+03:00" level=debug msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2681881324 localIndex=2681881324 remoteIndex=0 udpAddrs="[35.73.119.11:42420 172.31.42.166:42420]" vpnIp=10.145.0.41
...
time="2025-12-23T11:23:47+03:00" level=debug msg="Generated index" index=3582854421
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.145.0.41 localIndexId:3582854421] mapTotalSize:34 vpnIp:10.145.0.41]"
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message sent" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:2 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=3582854421 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=debug msg="Generated index" index=3676844372
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=debug msg="Generated index" index=186754654
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=debug msg="Generated index" index=3001560284
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message received" certName=t2-t2 fingerprint=7c541cb94aa12707cd7b513ccbd7e268c4b7a100a1a0de1444cb4e6e9617b96e handshake="map[stage:1 style:ix_psk0]" initiatorIndex=206017903 issuer=87cb12905045a4f023ce38a6f49fed8430a94283b58dc0f88ce7bd36bacd617a remoteIndex=0 responderIndex=0 udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41
time="2025-12-23T11:23:47+03:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" udpAddr="35.73.119.11:42420" vpnIp=10.145.0.41

[JackDoan]

Hey @timteka, sorry to keep you hanging, hope you enjoyed the holidays!

We even made firewall holes for incoming 4242. Maybe it's also not correct. Do ordinary nodes need to have any incoming ports specifically opened? Or only the lighthouses, sitting on white ips, and relays? And what if ee opened some incoming ports like 4242, could we've broken "normal" way of things?

In general, I've had the best results by configuring AWS to allow UDP 4242 (or your Nebula port of choice) from everywhere that makes sense. Usually that's 0.0.0.0/0. Nebula can hole-punch, so you don't always need that rule, but then you're relying stuff not entirely under your control, like conntrack behavior. Also, in my experience at least, AWS's firewall is "symmetric", meaning that it will only accept UDP reply traffic from the hosts your host has sent traffic to (unless you open the port).

TLDR, yes, allow UDP 4242 inbound as much as possible for best results.

[timteka]

@JackDoan thank you so much. Opening inbound 4242 solved the problem