Connectivity issue when pod A send traffic to service which points to pod A back #648
Description
When a pod sends traffic to the exact same pod itself via a service, the traffic will fail.
What happened:
There is repro and log provided. Here is the analysis.
When a pod tries to connect to the pod itself, it's a direct connection and actually mizar is not involved. The connection will succeed because mizar will bypass the traffic.
But a pod may connect to a kubernetes service, and this service may actually pointing to the exact same pod. In this scenario, the pod sends traffic to service ip, and mizar networking is involved to handle the traffic. Then mizar will figure out from service ip to the real destination ip behind. Then mizar will replace the destination ip from the service ip to the destination ip.
In the issue's scenario, the traffic is from a pod, with ip (for example 11.0.0.11). And the service ip will point to the same destination pod, which the ip is 11.0.0.11. Now the packet will look as [source ip: 11.0.0.11, destination ip: 11.0.0.11]. The source and destination ip are the same. I believe mizar didn't aware such packet will occur then has no way to handle it for now.
How to reproduce it (as minimally and precisely as possible):
- Start a pod
kubectl run pod1 --image=fwnetworking/testpod - Create a service which pointing to the pod
cat > /home/ubuntu/go/src/k8s.io/mizar/mizar/obj/tests/test_service.yaml
apiVersion: v1
kind: Service
metadata:
name: test-service-3
annotations:
service.beta.kubernetes.io/mizar-scaled-endpoint-type: "scaled-endpoint"
labels:
run: test-service-3
spec:
ports:- name: http
protocol: TCP
port: 8000
targetPort: 8000
selector:
run: pod1
- name: http
(ctrl+D)
kubectl apply -f /home/ubuntu/go/src/k8s.io/mizar/mizar/obj/tests/test_service.yaml
3. Get ip for the pod and the service
kubectl get pods -owide
kubectl get services
5. Go into the pod
kubectl exec -i -t pod1 -- /bin/bash
6. Send traffic
Send traffic to the pod ip directly, succeed.
curl [pod ip]:8000
Send traffic to the service ip, failed.
curl [service ip]: 8000
Anything else we need to know?:
Network tracing log:
root@ip-172-31-20-85:/sys/kernel/tracing# cat trace_pipe | grep -i 0000b9 -A100 -B20
coredns-75061 [005] d.s1 2447.601923: bpf_trace_printk: Modified IP Address, src: 0x55141fac, dst: 0x55141fac, csum: 0x3dfa
<idle>-0 [007] d.s. 2448.270493: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.279005: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.287012: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.294925: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.302724: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.310856: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2448.318952: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
curl-115336 [001] d.s1 2448.645102: bpf_trace_printk: [Agent:2580842.0xb00000b]
curl-115336 [001] d.s1 2448.645149: bpf_trace_printk: [Agent:2580842.0xb00000b] Processing IP
curl-115336 [001] d.s1 2448.645155: bpf_trace_printk: [Agent:2580842.0xa0000b9] No dest IP address found! [473]
curl-115336 [001] d.s1 2448.645158: bpf_trace_printk: [Agent:2580842.0xb00000b] Sending dst 0xa0000b9, to transit switch!
curl-115336 [001] d.s1 2448.645162: bpf_trace_printk: [Agent:2580842.0xb00000b] TAILCALL: transit switch on same host. Tunnel to dst=[0xac1f1455].
curl-115336 [001] d.s1 2448.645164: bpf_trace_printk: [Transit:728:0xac1f1455] Processing IP
curl-115336 [001] d.s1 2448.645168: bpf_trace_printk: [Transit::0xac1f1455] RX: {src=0xac1f1455, dst=0xac1f1455}/
curl-115336 [001] d.s1 2448.645169: bpf_trace_printk: [Transit::0xac1f1455] RX: {vni:0x27616a}/
curl-115336 [001] d.s1 2448.645170: bpf_trace_printk: [Transit::0xac1f1455] RX: {in.src=0xb00000b, in.dst=0xa0000b9}
curl-115336 [001] d.s1 2448.645171: bpf_trace_printk: [Transit:311:] This is a scaled endpoint, the transit switch will handle it!
curl-115336 [001] d.s1 2448.645173: bpf_trace_printk: [Scaled_EP:480:0xac1f1455] Process TCP
curl-115336 [001] d.s1 2448.645183: bpf_trace_printk: Modified Inner IP Address, src: 0xb00000b, dst: 0xb00000b, csum: 0x91c7
coredns-75051 [000] d.s1 2449.064635: bpf_trace_printk: [Agent:2580842.0xb000002]
coredns-75051 [000] d.s1 2449.064677: bpf_trace_printk: [Agent:2580842.0xb000002] Processing IP
coredns-75051 [000] d.s1 2449.064682: bpf_trace_printk: [Agent:2580842.0xac1f0002] No dest IP address found! [473]
coredns-75051 [000] d.s1 2449.064684: bpf_trace_printk: [Agent:2580842.0xb000002] Sending dst 0xac1f0002, to transit switch!
coredns-75051 [000] d.s1 2449.064688: bpf_trace_printk: [Agent:2580842.0xb000002] TAILCALL: transit switch on same host. Tunnel to dst=[0xac1f1455].
coredns-75051 [000] d.s1 2449.064689: bpf_trace_printk: [Transit:728:0xac1f1455] Processing IP
coredns-75051 [000] d.s1 2449.064690: bpf_trace_printk: [Scaled_EP:449:0xac1f1455] Process UDP
coredns-75051 [000] d.s1 2449.064693: bpf_trace_printk: [Transit::0xac1f1455] RX: {src=0xac1f1455, dst=0xac1f1455}/
coredns-75051 [000] d.s1 2449.064694: bpf_trace_printk: [Transit::0xac1f1455] RX: {vni:0x27616a}/
coredns-75051 [000] d.s1 2449.064695: bpf_trace_printk: [Transit::0xac1f1455] RX: {in.src=0xb000002, in.dst=0xac1f0002}
coredns-75051 [000] d.s1 2449.064696: bpf_trace_printk: [Transit::] LPM lookup key [0x60:0x2001fac]!
coredns-75051 [000] d.s1 2449.064699: bpf_trace_printk: [Transit:236:] Sending packet to router!
coredns-75051 [000] d.s1 2449.064700: bpf_trace_printk: Modified IP Address, src: 0x55141fac, dst: 0x55141fac, csum: 0x3dfa
<idle>-0 [007] d.s. 2449.329408: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2449.338006: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2449.346356: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2449.354310: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2449.362207: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
containerd-shim-72090 [007] d.s. 2449.370150: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [007] d.s. 2449.378126: bpf_trace_printk: [Transit:798:0xac1f1455] PASS non-geneve packet
<idle>-0 [001] d.s. 2449.666503: bpf_trace_printk: [Agent:2580842.0xb00000b]
<idle>-0 [001] d.s. 2449.666543: bpf_trace_printk: [Agent:2580842.0xb00000b] Processing IP
<idle>-0 [001] d.s. 2449.666549: bpf_trace_printk: Modified Inner TCP Ports src: 37832, dest: 8000, csum: 0xee65
<idle>-0 [001] d.s. 2449.666551: bpf_trace_printk: Modified Inner IP Address, src: 0xb00000b, dst: 0xb00000b, csum: 0x90c7
<idle>-0 [001] d.s. 2449.666553: bpf_trace_printk: [Agent:2580842.0xb00000b] Sending dst 0xb00000b, to transit switch!
<idle>-0 [001] dns. 2449.666578: bpf_trace_printk: [Agent:2580842.0xb00000b] TAILCALL: transit switch on same host. Tunnel to dst=[0xac1f1455].
<idle>-0 [001] dns. 2449.666579: bpf_trace_printk: [Transit:728:0xac1f1455] Processing IP
<idle>-0 [001] dNs. 2449.666588: bpf_trace_printk: [Transit::0xac1f1455] RX: {src=0xac1f1455, dst=0xac1f1455}/
<idle>-0 [001] dNs. 2449.666589: bpf_trace_printk: [Transit::0xac1f1455] RX: {vni:0x27616a}/
<idle>-0 [001] dNs. 2449.666590: bpf_trace_printk: [Transit::0xac1f1455] RX: {in.src=0xb00000b, in.dst=0xb00000b}
<idle>-0 [001] dNs. 2449.666592: bpf_trace_printk: [Transit::0xac1f1455] This is the ep host dst=[184549387] @ itf=[27]
<idle>-0 [001] dNs. 2449.666593: bpf_trace_printk: [Transit::0xac1f1455] REDIRECT: {src=0xac1f1455, dst=0xac1f1455}/
<idle>-0 [001] dNs. 2449.666593: bpf_trace_printk: [Transit::0xac1f1455] REDIRECT: {vni:0x27616a}/
<idle>-0 [001] dNs. 2449.666594: bpf_trace_printk: [Transit::0xac1f1455] REDIRECT: {in.src=0xb00000b, in.dst=0xb00000b}
<idle>-0 [001] dNs. 2449.666595: bpf_trace_printk: [Transit:128:0xac1f1455] REDIRECT: itf=[27].
hyperkube-62859 [006] d.s1 2450.309090: bpf_trace_printk: [Agent:2580842.0xac1f1455]
hyperkube-62859 [006] d.s1 2450.309137: bpf_trace_printk: [Agent:2580842.0xac1f1455] Processing IP
hyperkube-62859 [006] d.s1 2450.309143: bpf_trace_printk: [Agent:2580842.0xb000002] No dest IP address found! [473]
hyperkube-62859 [006] d.s1 2450.309145: bpf_trace_printk: [Agent:2580842.0xac1f1455] Sending dst 0xb000002, to transit switch!
hyperkube-62859 [006] d.s1 2450.309170: bpf_trace_printk: [Agent:2580842.0xac1f1455] TAILCALL: transit switch on same host. Tunnel to dst=[0xac1f1455].
hyperkube-62859 [006] d.s1 2450.309172: bpf_trace_printk: [Transit:728:0xac1f1455] Processing IP
hyperkube-62859 [006] d.s1 2450.309176: bpf_trace_printk: [Transit::0xac1f1455] RX: {src=0xac1f1455, dst=0xac1f1455}/
hyperkube-62859 [006] d.s1 2450.309177: bpf_trace_printk: [Transit::0xac1f1455] RX: {vni:0x27616a}/
hyperkube-62859 [006] d.s1 2450.309180: bpf_trace_printk: [Transit::0xac1f1455] RX: {in.src=0xac1f1455, in.dst=0xb000002}
hyperkube-62859 [006] d.s1 2450.309181: bpf_trace_printk: [Transit::0xac1f1455] This is the ep host dst=[184549378] @ itf=[15]
hyperkube-62859 [006] d.s1 2450.309181: bpf_trace_printk: [Transit::0xac1f1455] REDIRECT: {src=0xac1f1455, dst=0xac1f1455}/
hyperkube-62859 [006] d.s1 2450.309181: bpf_trace_printk: [Transit::0xac1f1455] REDIRECT: {vni:0x27616a}/