why my mayastor only have a couple of kB/s speed?

[13567436138]

fio --name=4k-random-read --filename=/data/testfile --rw=randread --bs=4k --direct=1 --size=100M --runtime=60 --group_reporting --end_fsync=0


fio-3.36
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3KiB/s][r=0 IOPS][eta 00m:00s]  
4k-random-read: (groupid=0, jobs=1): err= 0: pid=3379: Fri Oct 31 15:54:37 2025
  read: IOPS=4, BW=18.0KiB/s (18.4kB/s)(1080KiB/60130msec)
    clat (usec): min=1007, max=1707.7k, avg=222629.57, stdev=316259.83
     lat (usec): min=1007, max=1707.7k, avg=222630.46, stdev=316259.84
    clat percentiles (usec):
     |  1.00th=[   1090],  5.00th=[   3589], 10.00th=[   3982],
     | 20.00th=[   4015], 30.00th=[   7963], 40.00th=[  35390],
     | 50.00th=[  83362], 60.00th=[ 166724], 70.00th=[ 254804],
     | 80.00th=[ 392168], 90.00th=[ 683672], 95.00th=[ 926942],
     | 99.00th=[1451230], 99.50th=[1468007], 99.90th=[1702888],
     | 99.95th=[1702888], 99.99th=[1702888]
   bw (  KiB/s): min=    8, max=  272, per=100.00%, avg=22.87, stdev=32.31, samples=94
   iops        : min=    2, max=   68, avg= 5.70, stdev= 8.06, samples=94
  lat (msec)   : 2=2.96%, 4=16.67%, 10=12.96%, 20=1.85%, 50=8.52%
  lat (msec)   : 100=12.22%, 250=14.44%, 500=15.93%, 750=6.30%, 1000=4.81%
  lat (msec)   : 2000=3.33%
  cpu          : usr=0.00%, sys=0.06%, ctx=270, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=270,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=18.0KiB/s (18.4kB/s), 18.0KiB/s-18.0KiB/s (18.4kB/s-18.4kB/s), io=1080KiB (1106kB), run=60130-60130msec

Disk stats (read/write):
  nvme7n1: ios=279/541, sectors=2664/133496, merge=0/0, ticks=61336/100852, in_queue=162188, util=100.00%

[tiagolobocastro]

Without more details such as network speed, core isolation, native hard drive, etc it's hard to say

[13567436138]

hard disk have 60MB/s

fio --name=safe-test \
    --filename=/var/lib/libvirt/disk/cn-sh-1a/fio-testfile \
    --rw=randread \
    --bs=4k \
    --size=100M \
    --runtime=30 \
    --direct=1 \
    --group_reporting \
    --end_fsync=0
    
safe-test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.36
Starting 1 process
safe-test: Laying out IO file (1 file / 100MiB)
Jobs: 1 (f=1)
safe-test: (groupid=0, jobs=1): err= 0: pid=90688: Fri Oct 31 16:22:43 2025
  read: IOPS=14.5k, BW=56.7MiB/s (59.4MB/s)(100MiB/1765msec)
    clat (usec): min=50, max=690, avg=63.17, stdev=17.22
     lat (usec): min=51, max=693, avg=64.96, stdev=17.37
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   56], 10.00th=[   57], 20.00th=[   58],
     | 30.00th=[   60], 40.00th=[   61], 50.00th=[   62], 60.00th=[   63],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   70], 95.00th=[   75],
     | 99.00th=[   91], 99.50th=[  101], 99.90th=[  367], 99.95th=[  412],
     | 99.99th=[  515]
   bw (  KiB/s): min=57256, max=58976, per=99.85%, avg=57930.67, stdev=917.96, samples=3
   iops        : min=14314, max=14744, avg=14482.67, stdev=229.49, samples=3
  lat (usec)   : 100=99.45%, 250=0.31%, 500=0.22%, 750=0.01%
  cpu          : usr=2.27%, sys=28.63%, ctx=25600, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=56.7MiB/s (59.4MB/s), 56.7MiB/s-56.7MiB/s (59.4MB/s-59.4MB/s), io=100MiB (105MB), run=1765-1765msec

Disk stats (read/write):
  nvme2n1: ios=24953/197, sectors=199624/1504, merge=0/0, ticks=1153/49, in_queue=1243, util=61.75%

[13567436138]

change io-engine bind cpu from 10 to 2 ,it is faster,but not enouth

root@fio-test:/# fio --name=4k-random-read --filename=/data/testfile --rw=randread --bs=4k --direct=1 --size=100M --runtime=60 --group_reporting --end_fsync=0porting --end_fsync=0rea
4k-random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=4140KiB/s][r=1035 IOPS][eta 00m:00s]
4k-random-read: (groupid=0, jobs=1): err= 0: pid=3362: Sat Nov  1 12:26:29 2025
  read: IOPS=1052, BW=4210KiB/s (4311kB/s)(100MiB/24321msec)
    clat (usec): min=483, max=22700, avg=945.52, stdev=524.50
     lat (usec): min=484, max=22700, avg=946.42, stdev=524.52
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  685], 10.00th=[  717], 20.00th=[  775],
     | 30.00th=[  832], 40.00th=[  873], 50.00th=[  906], 60.00th=[  938],
     | 70.00th=[  971], 80.00th=[ 1012], 90.00th=[ 1074], 95.00th=[ 1156],
     | 99.00th=[ 2638], 99.50th=[ 4015], 99.90th=[ 7963], 99.95th=[10945],
     | 99.99th=[20841]
   bw (  KiB/s): min= 3656, max= 4912, per=100.00%, avg=4216.98, stdev=278.28, samples=48
   iops        : min=  914, max= 1228, avg=1054.23, stdev=69.57, samples=48
  lat (usec)   : 500=0.01%, 750=15.04%, 1000=63.30%
  lat (msec)   : 2=20.43%, 4=0.71%, 10=0.46%, 20=0.05%, 50=0.01%
  cpu          : usr=0.54%, sys=4.21%, ctx=25600, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=4210KiB/s (4311kB/s), 4210KiB/s-4210KiB/s (4311kB/s-4311kB/s), io=100MiB (105MB), run=24321-24321msec

Disk stats (read/write):
  nvme1n1: ios=25402/4, sectors=203208/40, merge=0/0, ticks=22691/5, in_queue=22696, util=93.77%

[13567436138]

network between node,is fast

[root@kamaji-manager-01-c ~]# iperf3 -c 192.168.10.115 -t 30 -P 4
Connecting to host 192.168.10.115, port 5201
[  5] local 192.168.10.116 port 53618 connected to 192.168.10.115 port 5201
[  7] local 192.168.10.116 port 53626 connected to 192.168.10.115 port 5201
[  9] local 192.168.10.116 port 53632 connected to 192.168.10.115 port 5201
[ 11] local 192.168.10.116 port 53636 connected to 192.168.10.115 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   747 MBytes  6.26 Gbits/sec    0   4.06 MBytes       
[  7]   0.00-1.00   sec   768 MBytes  6.44 Gbits/sec    0   3.94 MBytes       
[  9]   0.00-1.00   sec   758 MBytes  6.36 Gbits/sec    0   4.15 MBytes       
[ 11]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0   4.06 MBytes       
[SUM]   0.00-1.00   sec  2.95 GBytes  25.3 Gbits/sec    0             

[13567436138]

pod service network is not slow

[root@kamaji-manager-01-a ~]# kubectl logs -f iperf3-client-jrfc9 
{
        "start":        {
                "connected":    [{
                                "socket":       5,
                                "local_host":   "10.248.213.101",
                                "local_port":   54894,
                                "remote_host":  "10.85.183.83",
                                "remote_port":  5201
                        }, {
                                "socket":       7,
                                "local_host":   "10.248.213.101",
                                "local_port":   54896,
                                "remote_host":  "10.85.183.83",
                                "remote_port":  5201
                        }, {
                                "socket":       9,
                                "local_host":   "10.248.213.101",
                                "local_port":   54906,
                                "remote_host":  "10.85.183.83",
                                "remote_port":  5201
                        }, {
                                "socket":       11,
                                "local_host":   "10.248.213.101",
                                "local_port":   54912,
                                "remote_host":  "10.85.183.83",
                                "remote_port":  5201
                        }],
                "version":      "iperf 3.12",
                "system_info":  "Linux iperf3-client-jrfc9 6.6.0-98.0.0.103.oe2403sp2.x86_64 #1 SMP Wed Jun 25 23:16:20 CST 2025 x86_64",
                "timestamp":    {
                        "time": "Sat, 01 Nov 2025 04:54:04 GMT",
                        "timesecs":     1761972844
                },
                "connecting_to":        {
                        "host": "iperf3-server",
                        "port": 5201
                },
                "cookie":       "lpqg2yvqxci2jcpp3d6twfe6y2eesjnyh222",
                "tcp_mss_default":      1398,
                "target_bitrate":       0,
                "sock_bufsize": 0,
                "sndbuf_actual":        16384,
                "rcvbuf_actual":        131072,
                "sock_bufsize": 0,
                "sndbuf_actual":        16384,
                "rcvbuf_actual":        131072,
                "sock_bufsize": 0,
                "sndbuf_actual":        16384,
                "rcvbuf_actual":        131072,
                "sock_bufsize": 0,
                "sndbuf_actual":        16384,
                "rcvbuf_actual":        131072,
                "test_start":   {
                        "protocol":     "TCP",
                        "num_streams":  4,
                        "blksize":      131072,
                        "omit": 0,
                        "duration":     30,
                        "bytes":        0,
                        "blocks":       0,
                        "reverse":      0,
                        "tos":  0,
                        "target_bitrate":       0
                }
        },
        "intervals":    [{
                        "streams":      [{
                                        "socket":       5,
                                        "start":        0,
                                        "end":  1.001849,
                                        "seconds":      1.0018490552902222,
                                        "bytes":        461373440,
                                        "bits_per_second":      3684175276.2154083,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5523712,
                                        "rtt":  36,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        0,
                                        "end":  1.001855,
                                        "seconds":      1.0018550157547,
                                        "bytes":        461373440,
                                        "bits_per_second":      3684153357.4791465,
                                        "retransmits":  0,
                                        "snd_cwnd":     274008,
                                        "snd_wnd":      2272896,
                                        "rtt":  38,
                                        "rttvar":       12,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        0,
                                        "end":  1.001856,
                                        "seconds":      1.0018559694290161,
                                        "bytes":        461373440,
                                        "bits_per_second":      3684149850.5055475,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5523712,
                                        "rtt":  34,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        0,
                                        "end":  1.001857,
                                        "seconds":      1.0018570423126221,
                                        "bytes":        460062720,
                                        "bits_per_second":      3673679581.5740013,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      2176000,
                                        "rtt":  36,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        0,
                                "end":  1.001849,
                                "seconds":      1.0018490552902222,
                                "bytes":        1844183040,
                                "bits_per_second":      14726234697.826929,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        1.001849,
                                        "end":  2.000179,
                                        "seconds":      0.998329997062683,
                                        "bytes":        560988160,
                                        "bits_per_second":      4495412632.3003931,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5526656,
                                        "rtt":  35,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        1.001855,
                                        "end":  2.000186,
                                        "seconds":      0.99833101034164429,
                                        "bytes":        560988160,
                                        "bits_per_second":      4495408069.5782146,
                                        "retransmits":  0,
                                        "snd_cwnd":     274008,
                                        "snd_wnd":      2272896,
                                        "rtt":  35,
                                        "rttvar":       2,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        1.001856,
                                        "end":  2.000188,
                                        "seconds":      0.99833202362060547,
                                        "bytes":        560988160,
                                        "bits_per_second":      4495403506.8652983,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5523712,
                                        "rtt":  34,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        1.001857,
                                        "end":  2.000198,
                                        "seconds":      0.99834102392196655,
                                        "bytes":        560988160,
                                        "bits_per_second":      4495362979.6453085,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      2230400,
                                        "rtt":  38,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        1.001849,
                                "end":  2.000179,
                                "seconds":      0.998329997062683,
                                "bytes":        2243952640,
                                "bits_per_second":      17981650529.201572,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        2.000179,
                                        "end":  3.001022,
                                        "seconds":      1.0008430480957031,
                                        "bytes":        616038400,
                                        "bits_per_second":      4924155899.7457743,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5526656,
                                        "rtt":  43,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        2.000186,
                                        "end":  3.001037,
                                        "seconds":      1.000851035118103,
                                        "bytes":        616038400,
                                        "bits_per_second":      4924116603.8444939,
                                        "retransmits":  0,
                                        "snd_cwnd":     274008,
                                        "snd_wnd":      2272896,
                                        "rtt":  51,
                                        "rttvar":       14,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        2.000188,
                                        "end":  3.00104,
                                        "seconds":      1.0008519887924194,
                                        "bytes":        614727680,
                                        "bits_per_second":      4913635077.98352,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5523712,
                                        "rtt":  51,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        2.000198,
                                        "end":  3.001042,
                                        "seconds":      1.0008440017700195,
                                        "bytes":        616038400,
                                        "bits_per_second":      4924151207.6648874,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      2230400,
                                        "rtt":  48,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        2.000179,
                                "end":  3.001022,
                                "seconds":      1.0008430480957031,
                                "bytes":        2462842880,
                                "bits_per_second":      19686146671.536827,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        3.001022,
                                        "end":  4.000041,
                                        "seconds":      0.99901902675628662,
                                        "bytes":        551682048,
                                        "bits_per_second":      4417790117.9020033,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5526656,
                                        "rtt":  36,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        3.001037,
                                        "end":  4.000047,
                                        "seconds":      0.99901002645492554,
                                        "bytes":        551682048,
                                        "bits_per_second":      4417829918.7461967,
                                        "retransmits":  0,
                                        "snd_cwnd":     274008,
                                        "snd_wnd":      2275840,
                                        "rtt":  49,
                                        "rttvar":       19,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        3.00104,
                                        "end":  4.000049,
                                        "seconds":      0.99900901317596436,
                                        "bytes":        551813120,
                                        "bits_per_second":      4418884015.8366356,
                                        "retransmits":  0,
                                        "snd_cwnd":     352296,
                                        "snd_wnd":      5523712,
                                        "rtt":  37,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        3.001042,
                                        "end":  4.00005,
                                        "seconds":      0.99900799989700317,
                                        "bytes":        551813120,
                                        "bits_per_second":      4418888497.8449936,
                                        "retransmits":  0,
                                        "snd_cwnd":     369072,
                                        "snd_wnd":      2230400,
                                        "rtt":  36,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        3.001022,
                                "end":  4.000041,
                                "seconds":      0.99901902675628662,
                                "bytes":        2206990336,
                                "bits_per_second":      17673259682.8781,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        4.000041,
                                        "end":  5.000009,
                                        "seconds":      0.99996799230575562,
                                        "bytes":        572784640,
                                        "bits_per_second":      4582423792.81966,
                                        "retransmits":  0,
                                        "snd_cwnd":     369072,
                                        "snd_wnd":      5526656,
                                        "rtt":  38,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        4.000047,
                                        "end":  5.000016,
                                        "seconds":      0.9999690055847168,
                                        "bytes":        572784640,
                                        "bits_per_second":      4582419149.40212,
                                        "retransmits":  25,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  36,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        4.000049,
                                        "end":  5.000018,
                                        "seconds":      0.9999690055847168,
                                        "bytes":        572784640,
                                        "bits_per_second":      4582419149.40212,
                                        "retransmits":  0,
                                        "snd_cwnd":     373266,
                                        "snd_wnd":      5523712,
                                        "rtt":  34,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        4.00005,
                                        "end":  5.00002,
                                        "seconds":      0.999970018863678,
                                        "bytes":        572784640,
                                        "bits_per_second":      4582414505.993988,
                                        "retransmits":  46,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2701568,
                                        "rtt":  34,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        4.000041,
                                "end":  5.000009,
                                "seconds":      0.99996799230575562,
                                "bytes":        2291138560,
                                "bits_per_second":      18329695171.278637,
                                "retransmits":  71,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        5.000009,
                                        "end":  6.001392,
                                        "seconds":      1.0013829469680786,
                                        "bytes":        536084480,
                                        "bits_per_second":      4282753019.6963816,
                                        "retransmits":  0,
                                        "snd_cwnd":     369072,
                                        "snd_wnd":      5526656,
                                        "rtt":  41,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        5.000016,
                                        "end":  6.001396,
                                        "seconds":      1.00137996673584,
                                        "bytes":        536084480,
                                        "bits_per_second":      4282765765.7059321,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  38,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        5.000018,
                                        "end":  6.001417,
                                        "seconds":      1.001399040222168,
                                        "bytes":        536084480,
                                        "bits_per_second":      4282684192.5557714,
                                        "retransmits":  0,
                                        "snd_cwnd":     373266,
                                        "snd_wnd":      5523712,
                                        "rtt":  41,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        5.00002,
                                        "end":  6.001419,
                                        "seconds":      1.001399040222168,
                                        "bytes":        536084480,
                                        "bits_per_second":      4282684192.5557714,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2704512,
                                        "rtt":  38,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        5.000009,
                                "end":  6.001392,
                                "seconds":      1.0013829469680786,
                                "bytes":        2144337920,
                                "bits_per_second":      17131012078.785526,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        6.001392,
                                        "end":  7.000375,
                                        "seconds":      0.99898302555084229,
                                        "bytes":        553123840,
                                        "bits_per_second":      4429495403.64817,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      5526656,
                                        "rtt":  43,
                                        "rttvar":       14,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        6.001396,
                                        "end":  7.000381,
                                        "seconds":      0.99898499250412,
                                        "bytes":        553123840,
                                        "bits_per_second":      4429486682.18532,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  42,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        6.001417,
                                        "end":  7.000383,
                                        "seconds":      0.99896597862243652,
                                        "bytes":        553123840,
                                        "bits_per_second":      4429570991.09823,
                                        "retransmits":  0,
                                        "snd_cwnd":     391440,
                                        "snd_wnd":      5523712,
                                        "rtt":  36,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        6.001419,
                                        "end":  7.000385,
                                        "seconds":      0.99896597862243652,
                                        "bytes":        553123840,
                                        "bits_per_second":      4429570991.09823,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2647936,
                                        "rtt":  41,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        6.001392,
                                "end":  7.000375,
                                "seconds":      0.99898302555084229,
                                "bytes":        2212495360,
                                "bits_per_second":      17717981614.592686,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        7.000375,
                                        "end":  8.000939,
                                        "seconds":      1.0005639791488647,
                                        "bytes":        609484800,
                                        "bits_per_second":      4873130056.258564,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      5526656,
                                        "rtt":  40,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        7.000381,
                                        "end":  8.000944,
                                        "seconds":      1.0005630254745483,
                                        "bytes":        609484800,
                                        "bits_per_second":      4873134701.02242,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  35,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        7.000383,
                                        "end":  8.000946,
                                        "seconds":      1.0005630254745483,
                                        "bytes":        609484800,
                                        "bits_per_second":      4873134701.02242,
                                        "retransmits":  0,
                                        "snd_cwnd":     391440,
                                        "snd_wnd":      5523712,
                                        "rtt":  38,
                                        "rttvar":       12,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        7.000385,
                                        "end":  8.000948,
                                        "seconds":      1.0005630254745483,
                                        "bytes":        609484800,
                                        "bits_per_second":      4873134701.02242,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2704512,
                                        "rtt":  30,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        7.000375,
                                "end":  8.000939,
                                "seconds":      1.0005639791488647,
                                "bytes":        2437939200,
                                "bits_per_second":      19492520225.034256,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        8.000939,
                                        "end":  9.000256,
                                        "seconds":      0.9993169903755188,
                                        "bytes":        644874240,
                                        "bits_per_second":      5162519970.82665,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      5526656,
                                        "rtt":  48,
                                        "rttvar":       24,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        8.000944,
                                        "end":  9.000263,
                                        "seconds":      0.999319016933441,
                                        "bytes":        644874240,
                                        "bits_per_second":      5162509501.551506,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  52,
                                        "rttvar":       26,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        8.000946,
                                        "end":  9.000265,
                                        "seconds":      0.999319016933441,
                                        "bytes":        644874240,
                                        "bits_per_second":      5162509501.551506,
                                        "retransmits":  0,
                                        "snd_cwnd":     391440,
                                        "snd_wnd":      5526144,
                                        "rtt":  55,
                                        "rttvar":       28,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        8.000948,
                                        "end":  9.000266,
                                        "seconds":      0.99931800365448,
                                        "bytes":        644874240,
                                        "bits_per_second":      5162514736.18377,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2704512,
                                        "rtt":  52,
                                        "rttvar":       22,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        8.000939,
                                "end":  9.000256,
                                "seconds":      0.9993169903755188,
                                "bytes":        2579496960,
                                "bits_per_second":      20650079883.3066,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        9.000256,
                                        "end":  10.000939,
                                        "seconds":      1.0006829500198364,
                                        "bytes":        659292160,
                                        "bits_per_second":      5270737629.63129,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      5526656,
                                        "rtt":  42,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        9.000263,
                                        "end":  10.000962,
                                        "seconds":      1.0006990432739258,
                                        "bytes":        659292160,
                                        "bits_per_second":      5270652865.56513,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  36,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        9.000265,
                                        "end":  10.000962,
                                        "seconds":      1.0006970167160034,
                                        "bytes":        659292160,
                                        "bits_per_second":      5270663539.4086027,
                                        "retransmits":  0,
                                        "snd_cwnd":     391440,
                                        "snd_wnd":      5526656,
                                        "rtt":  42,
                                        "rttvar":       15,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        9.000266,
                                        "end":  10.000962,
                                        "seconds":      1.0006959438323975,
                                        "bytes":        659292160,
                                        "bits_per_second":      5270669190.2844143,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2704512,
                                        "rtt":  34,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        9.000256,
                                "end":  10.000939,
                                "seconds":      1.0006829500198364,
                                "bytes":        2637168640,
                                "bits_per_second":      21082950518.525162,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        10.000939,
                                        "end":  11.001808,
                                        "seconds":      1.0008690357208252,
                                        "bytes":        592445440,
                                        "bits_per_second":      4735448246.31983,
                                        "retransmits":  0,
                                        "snd_cwnd":     411012,
                                        "snd_wnd":      5526656,
                                        "rtt":  38,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        10.000962,
                                        "end":  11.001813,
                                        "seconds":      1.000851035118103,
                                        "bytes":        592445440,
                                        "bits_per_second":      4735533414.7610884,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  31,
                                        "rttvar":       3,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        10.000962,
                                        "end":  11.001815,
                                        "seconds":      1.0008529424667358,
                                        "bytes":        592445440,
                                        "bits_per_second":      4735524390.1453819,
                                        "retransmits":  0,
                                        "snd_cwnd":     411012,
                                        "snd_wnd":      5526656,
                                        "rtt":  33,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        10.000962,
                                        "end":  11.001817,
                                        "seconds":      1.0008549690246582,
                                        "bytes":        592445440,
                                        "bits_per_second":      4735514801.528882,
                                        "retransmits":  0,
                                        "snd_cwnd":     387246,
                                        "snd_wnd":      2704512,
                                        "rtt":  30,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        10.000939,
                                "end":  11.001808,
                                "seconds":      1.0008690357208252,
                                "bytes":        2369781760,
                                "bits_per_second":      18941792985.279316,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        11.001808,
                                        "end":  12.001158,
                                        "seconds":      0.99935001134872437,
                                        "bytes":        553123840,
                                        "bits_per_second":      4427868784.4592361,
                                        "retransmits":  0,
                                        "snd_cwnd":     411012,
                                        "snd_wnd":      5526656,
                                        "rtt":  44,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        11.001813,
                                        "end":  12.001164,
                                        "seconds":      0.99935102462768555,
                                        "bytes":        553123840,
                                        "bits_per_second":      4427864294.8793268,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  49,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        11.001815,
                                        "end":  12.001166,
                                        "seconds":      0.99935102462768555,
                                        "bytes":        553123840,
                                        "bits_per_second":      4427864294.8793268,
                                        "retransmits":  0,
                                        "snd_cwnd":     431982,
                                        "snd_wnd":      5526656,
                                        "rtt":  44,
                                        "rttvar":       14,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        11.001817,
                                        "end":  12.001168,
                                        "seconds":      0.99935102462768555,
                                        "bytes":        553123840,
                                        "bits_per_second":      4427864294.8793268,
                                        "retransmits":  0,
                                        "snd_cwnd":     390042,
                                        "snd_wnd":      2704512,
                                        "rtt":  42,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        11.001808,
                                "end":  12.001158,
                                "seconds":      0.99935001134872437,
                                "bytes":        2212495360,
                                "bits_per_second":      17711475137.836945,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        12.001158,
                                        "end":  13.000164,
                                        "seconds":      0.99900597333908081,
                                        "bytes":        496762880,
                                        "bits_per_second":      3978057335.0497046,
                                        "retransmits":  0,
                                        "snd_cwnd":     411012,
                                        "snd_wnd":      5526656,
                                        "rtt":  35,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        12.001164,
                                        "end":  13.000176,
                                        "seconds":      0.999011993408203,
                                        "bytes":        496762880,
                                        "bits_per_second":      3978033363.185215,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  38,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        12.001166,
                                        "end":  13.000178,
                                        "seconds":      0.999011993408203,
                                        "bytes":        496762880,
                                        "bits_per_second":      3978033363.185215,
                                        "retransmits":  0,
                                        "snd_cwnd":     431982,
                                        "snd_wnd":      5526656,
                                        "rtt":  42,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        12.001168,
                                        "end":  13.000217,
                                        "seconds":      0.99904900789260864,
                                        "bytes":        496762880,
                                        "bits_per_second":      3977885978.1693416,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  44,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        12.001158,
                                "end":  13.000164,
                                "seconds":      0.99900597333908081,
                                "bytes":        1987051520,
                                "bits_per_second":      15912229340.198818,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        13.000164,
                                        "end":  14.000719,
                                        "seconds":      1.0005550384521484,
                                        "bytes":        479723520,
                                        "bits_per_second":      3835659221.64265,
                                        "retransmits":  0,
                                        "snd_cwnd":     411012,
                                        "snd_wnd":      5526656,
                                        "rtt":  43,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        13.000176,
                                        "end":  14.000729,
                                        "seconds":      1.0005530118942261,
                                        "bytes":        479723520,
                                        "bits_per_second":      3835666990.5319457,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  38,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        13.000178,
                                        "end":  14.000732,
                                        "seconds":      1.0005539655685425,
                                        "bytes":        479723520,
                                        "bits_per_second":      3835663334.580122,
                                        "retransmits":  0,
                                        "snd_cwnd":     431982,
                                        "snd_wnd":      5526656,
                                        "rtt":  46,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        13.000217,
                                        "end":  14.000734,
                                        "seconds":      1.0005170106887817,
                                        "bytes":        479723520,
                                        "bits_per_second":      3835805007.8108792,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  35,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        13.000164,
                                "end":  14.000719,
                                "seconds":      1.0005550384521484,
                                "bytes":        1918894080,
                                "bits_per_second":      15342636886.5706,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        14.000719,
                                        "end":  15.000455,
                                        "seconds":      0.99973601102828979,
                                        "bytes":        470548480,
                                        "bits_per_second":      3765381859.2851291,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  48,
                                        "rttvar":       17,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        14.000729,
                                        "end":  15.000461,
                                        "seconds":      0.99973201751709,
                                        "bytes":        470548480,
                                        "bits_per_second":      3765396900.4105144,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  47,
                                        "rttvar":       17,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        14.000732,
                                        "end":  15.000463,
                                        "seconds":      0.99973100423812866,
                                        "bytes":        470548480,
                                        "bits_per_second":      3765400716.8345757,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  49,
                                        "rttvar":       13,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        14.000734,
                                        "end":  15.000465,
                                        "seconds":      0.99973100423812866,
                                        "bytes":        470548480,
                                        "bits_per_second":      3765400716.8345757,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2650112,
                                        "rtt":  45,
                                        "rttvar":       13,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        14.000719,
                                "end":  15.000455,
                                "seconds":      0.99973601102828979,
                                "bytes":        1882193920,
                                "bits_per_second":      15061527437.140516,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        15.000455,
                                        "end":  16.000292,
                                        "seconds":      0.99983698129653931,
                                        "bytes":        578027520,
                                        "bits_per_second":      4624974117.28414,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  34,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        15.000461,
                                        "end":  16.000302,
                                        "seconds":      0.99984097480773926,
                                        "bytes":        578027520,
                                        "bits_per_second":      4624955644.4605579,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  37,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        15.000463,
                                        "end":  16.000305,
                                        "seconds":      0.99984198808670044,
                                        "bytes":        578027520,
                                        "bits_per_second":      4624950957.3496876,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  35,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        15.000465,
                                        "end":  16.000307,
                                        "seconds":      0.99984198808670044,
                                        "bytes":        578027520,
                                        "bits_per_second":      4624950957.3496876,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  35,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        15.000455,
                                "end":  16.000292,
                                "seconds":      0.99983698129653931,
                                "bytes":        2312110080,
                                "bits_per_second":      18499896469.136555,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        16.000292,
                                        "end":  17.001264,
                                        "seconds":      1.0009720325469971,
                                        "bytes":        567541760,
                                        "bits_per_second":      4535925013.2563763,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  40,
                                        "rttvar":       13,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        16.000302,
                                        "end":  17.001273,
                                        "seconds":      1.0009709596633911,
                                        "bytes":        567541760,
                                        "bits_per_second":      4535929875.05535,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  33,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        16.000305,
                                        "end":  17.001293,
                                        "seconds":      1.0009880065917969,
                                        "bytes":        567541760,
                                        "bits_per_second":      4535852627.70441,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  33,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        16.000307,
                                        "end":  17.001296,
                                        "seconds":      1.0009889602661133,
                                        "bytes":        567541760,
                                        "bits_per_second":      4535848306.252,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  36,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        16.000292,
                                "end":  17.001264,
                                "seconds":      1.0009720325469971,
                                "bytes":        2270167040,
                                "bits_per_second":      18143700053.0255,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        17.001264,
                                        "end":  18.001746,
                                        "seconds":      1.0004819631576538,
                                        "bytes":        589824000,
                                        "bits_per_second":      4716318908.0465755,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  32,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        17.001273,
                                        "end":  18.001757,
                                        "seconds":      1.0004839897155762,
                                        "bytes":        589824000,
                                        "bits_per_second":      4716309354.7768126,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  34,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        17.001293,
                                        "end":  18.001762,
                                        "seconds":      1.0004689693450928,
                                        "bytes":        589824000,
                                        "bits_per_second":      4716380162.2840853,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  37,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        17.001296,
                                        "end":  18.001764,
                                        "seconds":      1.0004680156707764,
                                        "bytes":        589824000,
                                        "bits_per_second":      4716384658.0706139,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  35,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        17.001264,
                                "end":  18.001746,
                                "seconds":      1.0004819631576538,
                                "bytes":        2359296000,
                                "bits_per_second":      18865275632.1863,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        18.001746,
                                        "end":  19.000043,
                                        "seconds":      0.99829697608947754,
                                        "bytes":        567541760,
                                        "bits_per_second":      4548079568.25169,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  39,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        18.001757,
                                        "end":  19.00005,
                                        "seconds":      0.99829298257827759,
                                        "bytes":        567541760,
                                        "bits_per_second":      4548097762.1156282,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2275840,
                                        "rtt":  36,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        18.001762,
                                        "end":  19.000052,
                                        "seconds":      0.99829000234603882,
                                        "bytes":        567541760,
                                        "bits_per_second":      4548111339.7208767,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  37,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        18.001764,
                                        "end":  19.000054,
                                        "seconds":      0.99829000234603882,
                                        "bytes":        567541760,
                                        "bits_per_second":      4548111339.7208767,
                                        "retransmits":  0,
                                        "snd_cwnd":     392838,
                                        "snd_wnd":      2704512,
                                        "rtt":  37,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        18.001746,
                                "end":  19.000043,
                                "seconds":      0.99829697608947754,
                                "bytes":        2270167040,
                                "bits_per_second":      18192318273.006763,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        19.000043,
                                        "end":  20.001442,
                                        "seconds":      1.001399040222168,
                                        "bytes":        521666560,
                                        "bits_per_second":      4167501977.1080613,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  37,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        19.00005,
                                        "end":  20.001446,
                                        "seconds":      1.0013959407806396,
                                        "bytes":        521666560,
                                        "bits_per_second":      4167514876.0306263,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2701568,
                                        "rtt":  35,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        19.000052,
                                        "end":  20.001448,
                                        "seconds":      1.0013959407806396,
                                        "bytes":        521666560,
                                        "bits_per_second":      4167514876.0306263,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  33,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        19.000054,
                                        "end":  20.00145,
                                        "seconds":      1.0013959407806396,
                                        "bytes":        521666560,
                                        "bits_per_second":      4167514876.0306263,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2650112,
                                        "rtt":  35,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        19.000043,
                                "end":  20.001442,
                                "seconds":      1.001399040222168,
                                "bytes":        2086666240,
                                "bits_per_second":      16670007908.432245,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        20.001442,
                                        "end":  21.001179,
                                        "seconds":      0.999737024307251,
                                        "bytes":        496762880,
                                        "bits_per_second":      3975148407.4062181,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  49,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        20.001446,
                                        "end":  21.00126,
                                        "seconds":      0.999813973903656,
                                        "bytes":        496762880,
                                        "bits_per_second":      3974842464.42724,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2701568,
                                        "rtt":  39,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        20.001448,
                                        "end":  21.001264,
                                        "seconds":      0.99981600046157837,
                                        "bytes":        496762880,
                                        "bits_per_second":      3974834407.6963186,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  46,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        20.00145,
                                        "end":  21.001267,
                                        "seconds":      0.99981701374053955,
                                        "bytes":        496762880,
                                        "bits_per_second":      3974830379.3431058,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2704512,
                                        "rtt":  42,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        20.001442,
                                "end":  21.001179,
                                "seconds":      0.999737024307251,
                                "bytes":        1987051520,
                                "bits_per_second":      15900593629.624872,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        21.001179,
                                        "end":  22.002374,
                                        "seconds":      1.001194953918457,
                                        "bytes":        521666560,
                                        "bits_per_second":      4168351492.0510674,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  45,
                                        "rttvar":       19,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        21.00126,
                                        "end":  22.002383,
                                        "seconds":      1.0011229515075684,
                                        "bytes":        521666560,
                                        "bits_per_second":      4168651286.7530141,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  43,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        21.001264,
                                        "end":  22.002386,
                                        "seconds":      1.001121997833252,
                                        "bytes":        521666560,
                                        "bits_per_second":      4168655257.833137,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  37,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        21.001267,
                                        "end":  22.002388,
                                        "seconds":      1.0011210441589355,
                                        "bytes":        521666560,
                                        "bits_per_second":      4168659228.9208255,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2647936,
                                        "rtt":  47,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        21.001179,
                                "end":  22.002374,
                                "seconds":      1.001194953918457,
                                "bytes":        2086666240,
                                "bits_per_second":      16673405968.204269,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        22.002374,
                                        "end":  23.001568,
                                        "seconds":      0.99919402599334717,
                                        "bytes":        432537600,
                                        "bits_per_second":      3463091962.104104,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  42,
                                        "rttvar":       5,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        22.002383,
                                        "end":  23.001581,
                                        "seconds":      0.999198019504547,
                                        "bytes":        432537600,
                                        "bits_per_second":      3463078121.1073575,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  42,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        22.002386,
                                        "end":  23.001583,
                                        "seconds":      0.999197006225586,
                                        "bytes":        432537600,
                                        "bits_per_second":      3463081632.99158,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  45,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        22.002388,
                                        "end":  23.001586,
                                        "seconds":      0.999198019504547,
                                        "bytes":        432537600,
                                        "bits_per_second":      3463078121.1073575,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2650112,
                                        "rtt":  45,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        22.002374,
                                "end":  23.001568,
                                "seconds":      0.99919402599334717,
                                "bytes":        1730150400,
                                "bits_per_second":      13852367848.416416,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        23.001568,
                                        "end":  24.000825,
                                        "seconds":      0.99925702810287476,
                                        "bytes":        449576960,
                                        "bits_per_second":      3599289851.2090564,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  37,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        23.001581,
                                        "end":  24.000836,
                                        "seconds":      0.99925500154495239,
                                        "bytes":        449576960,
                                        "bits_per_second":      3599297150.8166156,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  41,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        23.001583,
                                        "end":  24.000853,
                                        "seconds":      0.99927002191543579,
                                        "bytes":        449576960,
                                        "bits_per_second":      3599243048.5464592,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  38,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        23.001586,
                                        "end":  24.000855,
                                        "seconds":      0.99926900863647461,
                                        "bytes":        449576960,
                                        "bits_per_second":      3599246698.25162,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2704512,
                                        "rtt":  44,
                                        "rttvar":       11,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        23.001568,
                                "end":  24.000825,
                                "seconds":      0.99925702810287476,
                                "bytes":        1798307840,
                                "bits_per_second":      14397159404.836226,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        24.000825,
                                        "end":  25.000484,
                                        "seconds":      0.99965900182724,
                                        "bytes":        440401920,
                                        "bits_per_second":      3524417179.8183622,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  47,
                                        "rttvar":       13,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        24.000836,
                                        "end":  25.000494,
                                        "seconds":      0.99965798854827881,
                                        "bytes":        440401920,
                                        "bits_per_second":      3524420752.257956,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  46,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        24.000853,
                                        "end":  25.000497,
                                        "seconds":      0.99964398145675659,
                                        "bytes":        440401920,
                                        "bits_per_second":      3524470136.7237811,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  42,
                                        "rttvar":       16,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        24.000855,
                                        "end":  25.000499,
                                        "seconds":      0.99964398145675659,
                                        "bytes":        440401920,
                                        "bits_per_second":      3524470136.7237811,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2704512,
                                        "rtt":  42,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        24.000825,
                                "end":  25.000484,
                                "seconds":      0.99965900182724,
                                "bytes":        1761607680,
                                "bits_per_second":      14097668719.273449,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        25.000484,
                                        "end":  26.000793,
                                        "seconds":      1.0003089904785156,
                                        "bytes":        473169920,
                                        "bits_per_second":      3784190081.2959862,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  39,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        25.000494,
                                        "end":  26.000803,
                                        "seconds":      1.0003089904785156,
                                        "bytes":        473169920,
                                        "bits_per_second":      3784190081.2959862,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  42,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        25.000497,
                                        "end":  26.000806,
                                        "seconds":      1.0003089904785156,
                                        "bytes":        473169920,
                                        "bits_per_second":      3784190081.2959862,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  41,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        25.000499,
                                        "end":  26.000808,
                                        "seconds":      1.0003089904785156,
                                        "bytes":        473169920,
                                        "bits_per_second":      3784190081.2959862,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2704512,
                                        "rtt":  41,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        25.000484,
                                "end":  26.000793,
                                "seconds":      1.0003089904785156,
                                "bytes":        1892679680,
                                "bits_per_second":      15136760325.183945,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        26.000793,
                                        "end":  27.002704,
                                        "seconds":      1.0019110441207886,
                                        "bytes":        484966400,
                                        "bits_per_second":      3872331004.5999122,
                                        "retransmits":  0,
                                        "snd_cwnd":     443166,
                                        "snd_wnd":      5526656,
                                        "rtt":  38,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        26.000803,
                                        "end":  27.002714,
                                        "seconds":      1.0019110441207886,
                                        "bytes":        484966400,
                                        "bits_per_second":      3872331004.5999122,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  43,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        26.000806,
                                        "end":  27.002718,
                                        "seconds":      1.001911997795105,
                                        "bytes":        484966400,
                                        "bits_per_second":      3872327318.7047119,
                                        "retransmits":  0,
                                        "snd_cwnd":     452952,
                                        "snd_wnd":      5526656,
                                        "rtt":  45,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        26.000808,
                                        "end":  27.002718,
                                        "seconds":      1.0019099712371826,
                                        "bytes":        484966400,
                                        "bits_per_second":      3872335151.2404,
                                        "retransmits":  0,
                                        "snd_cwnd":     394236,
                                        "snd_wnd":      2704512,
                                        "rtt":  43,
                                        "rttvar":       8,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        26.000793,
                                "end":  27.002704,
                                "seconds":      1.0019110441207886,
                                "bytes":        1939865600,
                                "bits_per_second":      15489324018.399649,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        27.002704,
                                        "end":  28.0028,
                                        "seconds":      1.0000959634780884,
                                        "bytes":        460062720,
                                        "bits_per_second":      3680148600.1404486,
                                        "retransmits":  0,
                                        "snd_cwnd":     465534,
                                        "snd_wnd":      5526656,
                                        "rtt":  41,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        27.002714,
                                        "end":  28.002859,
                                        "seconds":      1.0001449584960938,
                                        "bytes":        460062720,
                                        "bits_per_second":      3679968317.3270478,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  46,
                                        "rttvar":       13,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        27.002718,
                                        "end":  28.002865,
                                        "seconds":      1.0001469850540161,
                                        "bytes":        460062720,
                                        "bits_per_second":      3679960860.7541051,
                                        "retransmits":  0,
                                        "snd_cwnd":     475320,
                                        "snd_wnd":      5526656,
                                        "rtt":  39,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        27.002718,
                                        "end":  28.00289,
                                        "seconds":      1.0001720190048218,
                                        "bytes":        460062720,
                                        "bits_per_second":      3679868752.6392961,
                                        "retransmits":  0,
                                        "snd_cwnd":     409614,
                                        "snd_wnd":      2704512,
                                        "rtt":  44,
                                        "rttvar":       9,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        27.002704,
                                "end":  28.0028,
                                "seconds":      1.0000959634780884,
                                "bytes":        1840250880,
                                "bits_per_second":      14720594400.561794,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        28.0028,
                                        "end":  29.001901,
                                        "seconds":      0.99910098314285278,
                                        "bytes":        448266240,
                                        "bits_per_second":      3589356812.2805562,
                                        "retransmits":  0,
                                        "snd_cwnd":     465534,
                                        "snd_wnd":      5526656,
                                        "rtt":  43,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        28.002859,
                                        "end":  29.001913,
                                        "seconds":      0.99905401468276978,
                                        "bytes":        448266240,
                                        "bits_per_second":      3589525558.4741392,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  44,
                                        "rttvar":       6,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        28.002865,
                                        "end":  29.001916,
                                        "seconds":      0.99905097484588623,
                                        "bytes":        448266240,
                                        "bits_per_second":      3589536480.41152,
                                        "retransmits":  0,
                                        "snd_cwnd":     475320,
                                        "snd_wnd":      5526656,
                                        "rtt":  45,
                                        "rttvar":       4,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        28.00289,
                                        "end":  29.001917,
                                        "seconds":      0.99902701377868652,
                                        "bytes":        448266240,
                                        "bits_per_second":      3589622573.30354,
                                        "retransmits":  0,
                                        "snd_cwnd":     409614,
                                        "snd_wnd":      2704512,
                                        "rtt":  50,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        28.0028,
                                "end":  29.001901,
                                "seconds":      0.99910098314285278,
                                "bytes":        1793064960,
                                "bits_per_second":      14357427249.122225,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }, {
                        "streams":      [{
                                        "socket":       5,
                                        "start":        29.001901,
                                        "end":  30.000577,
                                        "seconds":      0.99867600202560425,
                                        "bytes":        470548480,
                                        "bits_per_second":      3769378489.4848089,
                                        "retransmits":  0,
                                        "snd_cwnd":     465534,
                                        "snd_wnd":      5526656,
                                        "rtt":  44,
                                        "rttvar":       14,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       7,
                                        "start":        29.001913,
                                        "end":  30.000588,
                                        "seconds":      0.998674988746643,
                                        "bytes":        470548480,
                                        "bits_per_second":      3769382313.9842339,
                                        "retransmits":  0,
                                        "snd_cwnd":     286590,
                                        "snd_wnd":      2704512,
                                        "rtt":  39,
                                        "rttvar":       10,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       9,
                                        "start":        29.001916,
                                        "end":  30.00059,
                                        "seconds":      0.998673975467682,
                                        "bytes":        470548480,
                                        "bits_per_second":      3769386138.49142,
                                        "retransmits":  0,
                                        "snd_cwnd":     475320,
                                        "snd_wnd":      5526656,
                                        "rtt":  39,
                                        "rttvar":       7,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }, {
                                        "socket":       11,
                                        "start":        29.001917,
                                        "end":  30.000592,
                                        "seconds":      0.998674988746643,
                                        "bytes":        470548480,
                                        "bits_per_second":      3769382313.9842339,
                                        "retransmits":  0,
                                        "snd_cwnd":     409614,
                                        "snd_wnd":      2704512,
                                        "rtt":  47,
                                        "rttvar":       17,
                                        "pmtu": 1450,
                                        "omitted":      false,
                                        "sender":       true
                                }],
                        "sum":  {
                                "start":        29.001901,
                                "end":  30.000577,
                                "seconds":      0.99867600202560425,
                                "bytes":        1882193920,
                                "bits_per_second":      15077513957.939236,
                                "retransmits":  0,
                                "omitted":      false,
                                "sender":       true
                        }
                }],
        "end":  {
                "streams":      [{
                                "sender":       {
                                        "socket":       5,
                                        "start":        0,
                                        "end":  30.000577,
                                        "seconds":      30.000577,
                                        "bytes":        15860891648,
                                        "bits_per_second":      4229489758.94697,
                                        "retransmits":  0,
                                        "max_snd_cwnd": 465534,
                                        "max_snd_wnd":  5526656,
                                        "max_rtt":      49,
                                        "min_rtt":      32,
                                        "mean_rtt":     40,
                                        "sender":       true
                                },
                                "receiver":     {
                                        "socket":       5,
                                        "start":        0,
                                        "end":  30.000663,
                                        "seconds":      30.000577,
                                        "bytes":        15860891648,
                                        "bits_per_second":      4229477634.6776071,
                                        "sender":       true
                                }
                        }, {
                                "sender":       {
                                        "socket":       7,
                                        "start":        0,
                                        "end":  30.000577,
                                        "seconds":      30.000577,
                                        "bytes":        15860891648,
                                        "bits_per_second":      4229489758.94697,
                                        "retransmits":  25,
                                        "max_snd_cwnd": 286590,
                                        "max_snd_wnd":  2704512,
                                        "max_rtt":      52,
                                        "min_rtt":      31,
                                        "mean_rtt":     40,
                                        "sender":       true
                                },
                                "receiver":     {
                                        "socket":       7,
                                        "start":        0,
                                        "end":  30.000663,
                                        "seconds":      30.000577,
                                        "bytes":        15860891648,
                                        "bits_per_second":      4229477634.6776071,
                                        "sender":       true
                                }
                        }, {
                                "sender":       {
                                        "socket":       9,
                                        "start":        0,
                                        "end":  30.000577,
                                        "seconds":      30.000577,
                                        "bytes":        15859712000,
                                        "bits_per_second":      4229175192.1971369,
                                        "retransmits":  0,
                                        "max_snd_cwnd": 475320,
                                        "max_snd_wnd":  5526656,
                                        "max_rtt":      55,
                                        "min_rtt":      33,
                                        "mean_rtt":     40,
                                        "sender":       true
                                },
                                "receiver":     {
                                        "socket":       9,
                                        "start":        0,
                                        "end":  30.000663,
                                        "seconds":      30.000577,
                                        "bytes":        15859712000,
                                        "bits_per_second":      4229163068.8295121,
                                        "sender":       true
                                }
                        }, {
                                "sender":       {
                                        "socket":       11,
                                        "start":        0,
                                        "end":  30.000577,
                                        "seconds":      30.000577,
                                        "bytes":        15859712000,
                                        "bits_per_second":      4229175192.1971369,
                                        "retransmits":  46,
                                        "max_snd_cwnd": 409614,
                                        "max_snd_wnd":  2704512,
                                        "max_rtt":      52,
                                        "min_rtt":      30,
                                        "mean_rtt":     40,
                                        "sender":       true
                                },
                                "receiver":     {
                                        "socket":       11,
                                        "start":        0,
                                        "end":  30.000663,
                                        "seconds":      30.000577,
                                        "bytes":        15859712000,
                                        "bits_per_second":      4229163068.8295121,
                                        "sender":       true
                                }
                        }],
                "sum_sent":     {
                        "start":        0,
                        "end":  30.000577,
                        "seconds":      30.000577,
                        "bytes":        63441207296,
                        "bits_per_second":      16917329902.288212,
                        "retransmits":  71,
                        "sender":       true
                },
                "sum_received": {
                        "start":        0,
                        "end":  30.000663,
                        "seconds":      30.000663,
                        "bytes":        63441207296,
                        "bits_per_second":      16917281407.014238,
                        "sender":       true
                },
                "cpu_utilization_percent":      {
                        "host_total":   99.955619969234249,
                        "host_user":    0.37288290473683261,
                        "host_system":  99.5827470622516,
                        "remote_total": 84.479945926573421,
                        "remote_user":  1.5412413208657081,
                        "remote_system":        82.938701272450828
                },
                "sender_tcp_congestion":        "cubic",
                "receiver_tcp_congestion":      "cubic"
        }
}
[root@kamaji-manager-01-a ~]# 

[tiagolobocastro]

Which cpus have you alloted for mayastor, are they isolated?
Please share a support bundle as well

[13567436138]

I don't know how to get support bundle,kubectl openebs command does not have this command.

[root@kamaji-manager-01-a ~]# kubectl openebs mayastor dump system  -n openebs
Failed to create loki client (Forward { source: ServiceNotFound { selector: "app=loki", namespace: NameSpace(Some("openebs")) } }). Continuing...
Label selectors : openebs.io/logging=true
Collecting logs...
         Collecting logs of service: csi-node, container: csi-driver-registrar of host: Some("kamaji-manager-01-b")
         Collecting logs of service: csi-controller, container: csi-controller of host: Some("kamaji-manager-01-b")
         Collecting logs of service: csi-node, container: csi-driver-registrar of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-zfs-controller, container: snapshot-controller of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-zfs-controller, container: openebs-zfs-plugin of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-zfs-node, container: openebs-zfs-plugin of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-zfs-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-zfs-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-lvm-controller, container: csi-resizer of host: Some("kamaji-manager-01-c")
         Collecting logs of service: operator-diskpool, container: operator-diskpool of host: Some("kamaji-manager-01-b")
Error fetching logs for pod : openebs-operator-diskpool-7dbf7d8f89-mb58q, error: K8sResourceError(ClientError(Api(ErrorResponse { status: "Failure", message: "previous terminated container \"operator-diskpool\" in pod \"openebs-operator-diskpool-7dbf7d8f89-mb58q\" not found", reason: "BadRequest", code: 400 })))
         Collecting logs of service: csi-node, container: csi-node of host: Some("kamaji-manager-01-c")
         Collecting logs of service: obs-callhome, container: obs-callhome of host: Some("kamaji-manager-01-c")
         Collecting logs of service: io-engine, container: metrics-exporter-io-engine of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-lvm-node, container: openebs-lvm-plugin of host: Some("kamaji-manager-01-d")
         Collecting logs of service: csi-controller, container: csi-snapshot-controller of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-zfs-controller, container: csi-resizer of host: Some("kamaji-manager-01-d")
         Collecting logs of service: csi-controller, container: csi-provisioner of host: Some("kamaji-manager-01-b")
         Collecting logs of service: agent-core, container: agent-core of host: Some("kamaji-manager-01-c")
         Collecting logs of service: io-engine, container: io-engine of host: Some("kamaji-manager-01-b")
         Collecting logs of service: io-engine, container: io-engine of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-lvm-controller, container: csi-snapshotter of host: Some("kamaji-manager-01-c")
         Collecting logs of service: csi-controller, container: csi-snapshotter of host: Some("kamaji-manager-01-b")
         Collecting logs of service: etcd, container: etcd of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-lvm-controller, container: snapshot-controller of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-lvm-controller, container: csi-provisioner of host: Some("kamaji-manager-01-c")
         Collecting logs of service: agent-core, container: agent-ha-cluster of host: Some("kamaji-manager-01-c")
         Collecting logs of service: agent-ha-node, container: agent-ha-node of host: Some("kamaji-manager-01-c")
         Collecting logs of service: api-rest, container: api-rest of host: Some("kamaji-manager-01-c")
         Collecting logs of service: etcd, container: etcd of host: Some("kamaji-manager-01-d")
         Collecting logs of service: localpv-provisioner, container: openebs-localpv-provisioner of host: Some("kamaji-manager-01-c")
         Collecting logs of service: io-engine, container: io-engine of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-lvm-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-zfs-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-zfs-controller, container: csi-provisioner of host: Some("kamaji-manager-01-d")
         Collecting logs of service: agent-ha-node, container: agent-ha-node of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-zfs-controller, container: csi-snapshotter of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-lvm-node, container: openebs-lvm-plugin of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-zfs-node, container: openebs-zfs-plugin of host: Some("kamaji-manager-01-c")
         Collecting logs of service: io-engine, container: metrics-exporter-io-engine of host: Some("kamaji-manager-01-c")
         Collecting logs of service: openebs-lvm-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-d")
         Collecting logs of service: csi-node, container: csi-node of host: Some("kamaji-manager-01-d")
         Collecting logs of service: io-engine, container: metrics-exporter-io-engine of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-lvm-node, container: openebs-lvm-plugin of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-zfs-node, container: openebs-zfs-plugin of host: Some("kamaji-manager-01-d")
         Collecting logs of service: csi-controller, container: csi-attacher of host: Some("kamaji-manager-01-b")
         Collecting logs of service: openebs-lvm-controller, container: openebs-lvm-plugin of host: Some("kamaji-manager-01-c")
         Collecting logs of service: agent-ha-node, container: agent-ha-node of host: Some("kamaji-manager-01-d")
         Collecting logs of service: openebs-lvm-node, container: csi-node-driver-registrar of host: Some("kamaji-manager-01-b")
         Collecting logs of service: csi-node, container: csi-driver-registrar of host: Some("kamaji-manager-01-c")
         Collecting logs of service: etcd, container: etcd of host: Some("kamaji-manager-01-b")
         Collecting logs of service: csi-controller, container: csi-resizer of host: Some("kamaji-manager-01-b")
         Collecting logs of service: csi-node, container: csi-node of host: Some("kamaji-manager-01-b")
Completed collection of logs
Collecting K8s resources...
         Collecting Kubernetes events
         Collecting daemonsets configuration
         Collecting deployments configuration
         Collecting statefulsets configuration
         Collecting Kubernetes pod resources
Completed collection of k8s resources
Collecting topology information...
Could not fetch rebuild history for 2af751dd-fecf-4314-8f0c-b10cf3935e42, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
Could not fetch rebuild history for 777d5118-426a-4daa-88d7-467accb225c7, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
         Collecting volume topology information
         Collecting snapshot topology information
         Collecting pool topology information
         Collecting node topology information
Completed collection of topology information
Collecting mayastor specific k8s resources...
         Collecting mayastor diskpool resources
         Collecting Kubernetes VolumeSnapshotClass resources
         Collecting Kubernetes VolumeSnapshotContents resources
Completed collection of mayastor specific k8s resources
Collecting mayastor specific information from Etcd...
         Failed to collect etcd dump information, error: Etcd(GetPrefix { prefix: "/openebs.io/mayastor/apis/v0/clusters/57a3083b-699e-434e-8153-c8512039e524/namespaces/openebs", source: GRpcStatus(Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }) })
Failed to dump system state
Failed to dump system state
Partially collected dump information: 
MultipleErrors([MultipleErrors([EtcdDumpError(Etcd(GetPrefix { prefix: "/openebs.io/mayastor/apis/v0/clusters/57a3083b-699e-434e-8153-c8512039e524/namespaces/openebs", source: GRpcStatus(Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }) }))])])

[tiagolobocastro]

You need to attach the tar file here

[13567436138]

how can you give commands

[tiagolobocastro]

Just attach the generated bundle tar.gz file here in a reply

[13567436138]

I have destroy the instance,maybe next time I installed ,I can give it

[13567436138]

cpu is not isolated,I use kvm vm to install k8s,and openebs is install in that k8s cluster

[13567436138]

@tiagolobocastro can you take a look

[13567436138]

I begin to install openebs again