Zero traffic in GTPUhw jumbo tests

[vrpolakatcisco]

Contrary to other jumbo issues, this is not specific to dev_iavf plugin. Also, GTPUsw tests are passing also with jumbo.
Telemetry [0] shows symptoms, first fragmentation happens, then the fragments are dropped instead of reassemblied. The latter could be explained by the fragmentation happening at unexpected step of the encap processing.
Comparing packet trace between hw [1] and sw [2], and ignoring irrelevant details like hashes and source ports, the processing is the same up to and including ip4-load-balance, but starts differing from ip4-rewrite on. Here are the relevant blocks:
SW:

00:00:23:513028: gtpu4-encap
  GTPU encap to gtpu_tunnel0 tteid 10 no-pdu-extension 
00:00:23:513029: ip4-load-balance
  fib 3 dpo-idx 5 flow hash: 0xfbb82cae
  UDP: 1.1.1.2 -> 1.1.1.1
    tos 0x00, ttl 254, length 9018, checksum 0x95ae dscp CS0 ecn NON_ECN
    fragment id 0x0000
  UDP: 44588 -> 2152
    length 8998, checksum 0x0000
00:00:23:513030: ip4-rewrite
  tx_sw_if_index 2 dpo-idx 5 : ipv4 via 1.1.1.1 HundredGigabitEthernet17/0/1: mtu:9145 next:4 flags:[] 40a6b7672750b49691b2a7b10800 flow hash: 0xfbb82cae
  00000000: 40a6b7672750b49691b2a7b108004500233a00000000fd1196ae010101020101
  00000020: 0101ae2c08682326000030ff23160000000a45002316000100003f3d
00:00:23:513030: HundredGigabitEthernet17/0/1-output

HW:

00:00:24:527955: gtpu4-encap
  GTPU encap to gtpu_tunnel0 tteid 10 no-pdu-extension 
00:00:24:527956: ip4-load-balance
  fib 3 dpo-idx 5 flow hash: 0xac1b271a
  UDP: 1.1.1.2 -> 1.1.1.1
    tos 0x00, ttl 254, length 9018, checksum 0x95ae dscp CS0 ecn NON_ECN
    fragment id 0x0000
  UDP: 6695 -> 2152
    length 8998, checksum 0x0000
00:00:24:527957: ip4-rewrite
  tx_sw_if_index 3 dpo-idx 5 : ipv4 via 1.1.1.1 HundredGigabitEthernet17/0/1: mtu:9000 next:4 flags:[] 40a6b7672750b49691b2a7b10800 flow hash: 0x00002328
  00000000: 4500233a00000000fe1195ae01010102010101011a2708682326000030ff2316
  00000020: 0000000a45002316000100003f3d1c850a0a0a061414140256666952
00:00:24:527959: ip4-frag

In both logs, tx_sw_if_index 2 is associated with a hardware interface where CSIT has set large enough MTU to fit 9000-byte plain packet plus encapsulation overhead, but tx_sw_if_index 3 still has the default MTU of 9000. The difference is only on the HW version not being simple enough to avoid software interface after encapsulation.

[0] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2510-3n-icx/15/log.html.gz#s1-s1-s1-s1-s1-t6-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1
[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-3n-icx/1753/log.html.gz#s1-s1-s1-s1-s1-t1-k3-k1-k1-k1-k1-k8-k14-k2-k1-k1-k1-k1
[2] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-3n-icx/1749/log.html.gz#s1-s1-s1-s1-s2-t1-k3-k1-k1-k1-k1-k8-k14-k2-k1-k1-k1-k1

[vrpolakatcisco]

The easiest solution to re-enable direct GTPUhw testing without fragmentation is for CSIT to increase the default MTU for software interfaces in VPP, but the currently available via startup.conf is not good enough. I mean I verified that when paired, a VPP change [3] and a CSIT change [4] can make the test pass, but I do not see an easy way to make the CSIT change is compatible with older VPP builds (for bisects, resets and other RCA activities). Older VPP builds exit with error when faced with default MTU value larger than 9000. A better fix would be to introduce a binary API so the CSIT config step failure is less fatal. I will open a separate VPP ticket for that.

[3] https://gerrit.fd.io/r/c/vpp/+/44211
[4] https://gerrit.fd.io/r/c/csit/+/44210

[vrpolakatcisco]

introduce a binary API

I did that in [5] (CSIT usage in [6]), but Damjan did not like that.

Then I finally spotted sw_interface_set_mtu API already exists and created CSIT-only change [7] to use it.
During debugging I confirmed it avoids vpp#3647 and that the fragmentation issue is caused by gtpu_offload_rx API refreshing [8] adjacency MTU for some reason. I guess during GTPU interface creating the code takes the generic L3 MTU (from maximum frame size CSIT already sets), but when flow offload is activated the code knows the protocol is IP4 (not updated by hw_interface_set_mtu).

Anyway, [7] solves this symptom of unintended fragmentation. The fact the traffic is still zero (now due to tx_errors) will be tracked in a new ticket.

[5] https://gerrit.fd.io/r/c/vpp/+/44236
[6] https://gerrit.fd.io/r/c/csit/+/44239
[7] https://gerrit.fd.io/r/c/csit/+/44264
[8] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-3n-icx/19927122591/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k8-k17-k1