3n-icx: regression in ip4tcpscale1cl10s-ldpreload-iperf3

[vrpolakatcisco]

After retest (rls 2506 CSIT code with 25.02-release VPP), only one tests shows big enough performance difference. Both [0] NICs [1] on 3n-icx show ~50% performance decrease, but 3na-spr [2] only ~10%. In all cases, standard deviation is too high for bisect job to converge into anything reasonable. Trending [3] is dominated by CSIT changes, so I am not optimistic about finding the cause.

[0] release comparison 3n-icx E810
[1] release comparison 3n-icx CX6
[2] release comparison 3na-spr CX7
[3] all 3 in trending

[vrpolakatcisco]

I created edited CSIT code [4] that repeats the robot test multiple times to get somewhat stable results, and this time bisect converged to 42544 [5]. I am not sure I trust all the classifications along the way (there might have been multiple anomalies) but I think this one culprit is likely guilty.

[4] 43641: Experiment: multiple robots per bisect | https://gerrit.fd.io/r/c/csit/+/43641
[5] 42544: vcl: improve vpp detached handling for mq epoll apps | https://gerrit.fd.io/r/c/vpp/+/42544

[vrpolakatcisco]

This run [6] shows 42544 decreasing CX7 3n-icx performance from 26+-6 Gbps down to 16+-6 Gbps.

[6] https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3n-icx/81

[fcoras]

Hard to see how [5] would impact performance as it only deals with vcl being detached from vpp, which should not happen during a test. @vrpolakatcisco

[vrpolakatcisco]

should not happen during a test

Is there a counter to check?

Hard to see how [5] would impact performance

After [6] I triggered 3 more runs, here are some stats:

Run 81.
Difference of averages relative to parent: -31.169364300469297%
Jumpavg representation of parent group: AvgStdevStats(size=57, avg=25904997548.970474, stdev=6267708571.105003)
Jumpavg representation of current group: AvgStdevStats(size=59, avg=17830574490.904224, stdev=6115469726.619658)

Run 82.
Difference of averages relative to parent: -22.97036574274829%
Jumpavg representation of parent group: AvgStdevStats(size=57, avg=25068671412.416435, stdev=5589284300.957322)
Jumpavg representation of current group: AvgStdevStats(size=58, avg=19310305902.136597, stdev=6639999828.89029)

Run 83.
Difference of averages relative to parent: -11.90044430178668%
Jumpavg representation of parent group: AvgStdevStats(size=59, avg=22011046155.8398, stdev=6037705543.599821)
Jumpavg representation of current group: AvgStdevStats(size=58, avg=19391633867.82353, stdev=7009886897.748701)

Run 84.
Difference of averages relative to parent: -22.141370846246623%
Jumpavg representation of parent group: AvgStdevStats(size=60, avg=25013659957.187874, stdev=5145068819.641068)
Jumpavg representation of current group: AvgStdevStats(size=60, avg=19475292743.847813, stdev=6602921769.795286)

So, [5] definitely affects the performance, but not enough to explain 50% difference between releases. Comparing to [1], it seems like there was an earlier regression from 35 to 25 Gbps.

[fcoras]

VCL detaches only if VPP crashes, so the counter would be a crashed VPP/failed test. Tests in my testbeds (skx and spr) do not exhibit any drop, so I'm still convinced it's something else ...

[vrpolakatcisco]

earlier regression from 35 to 25 Gbps.

First bisect did not confirm this, but another bisect found a progression shortly before 2502 release (after branch cut). Its cause [7] looked reasonable, before I realized the bisect used CX6 (with dpdk plugin usind mlx5 driver, not ice).

I still need to locate and investigate a regression between (master branch version of) [7] and [5].

[7] 42321: dpdk: patch add to ice driver for flow action handles | https://gerrit.fd.io/r/c/vpp/+/42321

[vrpolakatcisco]

I have some news. Bad news is that 42274 [8] (which is the change in master that got cherry-picked to become [7]) does not change performance in master. Here is the copy of the important part of the huge log in [9]:

Difference of averages relative to parent: 0.37373556720299106%
Jumpavg representation of parent group: AvgStdevStats(size=175, avg=39241258834.43329, stdev=2948307194.4127727)
Jumpavg representation of current group: AvgStdevStats(size=175, avg=39387917375.71575, stdev=2898890704.1734343)

The good news is that I found and confirmed another regression, from 30+-6 to 24+-6 Gbps, this time the cause is 42458 [10], again the log [11] is huge but the interesting part is:

Difference of averages relative to parent: -22.150562340938862%
Jumpavg representation of parent group: AvgStdevStats(size=177, avg=30470557866.134315, stdev=6782280974.390578)
Jumpavg representation of current group: AvgStdevStats(size=178, avg=23721157950.364384, stdev=6231100993.184288)

For completeness, I should also find and confirm the cause for decrease from ~40 to ~30 Gbps.

[8] 42274: dpdk: patch add to ice driver for flow action handles | https://gerrit.fd.io/r/c/vpp/+/42274
[9] https://logs.fd.io/vex-yul-rot-jenkins-1/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3n-icx/85/console.log.gz
[10] 42458: vcl: improve vpp detatch handling | https://gerrit.fd.io/r/c/vpp/+/42458
[11] https://logs.fd.io/vex-yul-rot-jenkins-1/vpp-csit-verify-perf-master-ubuntu2404-x86_64-3n-icx/86/console.log.gz

[vrpolakatcisco]

Few more runs in vpp-csit-verify-perf-master-ubuntu2404-x86_64-3n-icx job (#87 and #88) and I think they prove the issue is in the two testbeds not having the same performance. The one with host addresses 10.30.51.78-80 gets ~40Gbps on old builds, 10.30.51.75-77 gets ~30Gbps. As the bisect job unreserves testbed during compilation, the 30<->40 jumps it sees are caused by testbed selection (whether another job has claimed .77), not by VPP code. I can use custom CSIT code to use only one testbed, but we should also make the testbeds equal again.