Multi-job based data verification

[minwooim]

See the first commit description for the background and details.

The following jobfile is an example for the multiple write job based data verification.
16 jobs are going to write[zeroes, uncor, trim] to the same area [0, 1M).

[global]
ioengine=io_uring_cmd
cmd_type=nvme
filename=/dev/ng0n1
size=1M
thread=1

norandommap

ignore_error=0x2281  # to mask uncorred (errored) offsets
iodepth=64

group_reporting=1

bs=4k
numjobs=4

verify=crc32
verify_header_seed=0
do_verify=1

verify_table_id=1

[trim]
rw=randtrim

[uncor]
rw=randwrite
write_mode=uncor

[write]
rw=randwrite

[zeroes]
rw=randwrite
write_mode=zeroes
nonvectored=0  ; to mask BAD ADDRESS error in kernel
                                       

After run this jobfile, we can see the READ(verify) phase were issued only just for 1MB(256 commands):

     issued rwts: total=256,3072,1024,0 short=0,0,0,0 dropped=0,0,0,0

[vincentkfu]

Do you believe that multi-job verification is a sound test? Once requests are submitted to the device, they can be completed in an arbitrary order. For even a single job we can run into problems with repeated writes to the same offset completing in an order different from how they were submitted, leading to verification errors.

For example see:

commit b5ab9ba8e73692360e8ea5a5587724ecf74270f4
Author: Ankit Kumar <ankit.kumar@samsung.com>
Date:   Thu Jan 30 00:04:58 2025 +0530

    verify: disable write sequence checks with norandommap and iodepth > 1

    With norandommap for async I/O engines specifying I/O depth > 1, it is
    possible that two or more writes with the same offset are queued at once.
    When fio tries to verify the block, it may find a numberio mismatch
    because the writes did not land on the media in the order that they were
    queued. Avoid these spurious failures by disabling sequence number
    checking. Users will still be able to enable sequence number checking
    if they explicitly set the verify_header_sequence option.

    fio -name=verify -ioengine=libaio -rw=randwrite -verify=sha512 -direct=1 \
    -iodepth=32 -filesize=16M -bs=512 -norandommap=1 -debug=io,verify

    Below is the truncated log for the above command demonstrating the issue.
    This includes extra log entries when write sequence number is saved and
    retrieved.

    set: io_u->numberio=28489, off=0x5f2400
    queue: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    set: io_u->numberio=28574, off=0x5f2400
    iolog: overlap 6235136/512, 6235136/512
    queue: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    complete: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    complete: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0

    retrieve: io_u->numberio=28574, off=0x5f2400
    queue: io_u 0x5b8039e1db40: off=0x5f2400,len=0x200,ddir=0,file=verify.0.0

    bad header numberio 28489, wanted 28574 at file verify.0.0 offset 6235136, length 512 (requested block: offset=6235136, length=512)

    Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>

[sitsofe]

@vincentkfu I would argue that if you have a single job that may issue concurrent overlapping writes and you want to verify the result then you should use serialize_overlap (and that would have also prevented the issue generated by the job in the commit you referenced). However your point stands - without a thing that can serialize ALL overlapping writes regardless of which concurrent job submitted them, you open yourself up to regions where you cannot know what value they have...

[minwooim]

Do you believe that multi-job verification is a sound test?

Yes, I do. dedicated offsets for a job limits the variority of commands. If fio can manages overlapping offset ranges during command submission/completions, multi-job based data verification can test data integrity.

Once requests are submitted to the device, they can be completed in an arbitrary order. For even a single job we can run into problems with repeated writes to the same offset completing in an order different from how they were submitted, leading to verification errors.

That's why this change plugs the incoming submissions if offsets are overlapped. So, only a single write command can enter to the device at a time.

For example see:

commit b5ab9ba8e73692360e8ea5a5587724ecf74270f4
Author: Ankit Kumar <ankit.kumar@samsung.com>
Date:   Thu Jan 30 00:04:58 2025 +0530

    verify: disable write sequence checks with norandommap and iodepth > 1

    With norandommap for async I/O engines specifying I/O depth > 1, it is
    possible that two or more writes with the same offset are queued at once.
    When fio tries to verify the block, it may find a numberio mismatch
    because the writes did not land on the media in the order that they were
    queued. Avoid these spurious failures by disabling sequence number
    checking. Users will still be able to enable sequence number checking
    if they explicitly set the verify_header_sequence option.

    fio -name=verify -ioengine=libaio -rw=randwrite -verify=sha512 -direct=1 \
    -iodepth=32 -filesize=16M -bs=512 -norandommap=1 -debug=io,verify

    Below is the truncated log for the above command demonstrating the issue.
    This includes extra log entries when write sequence number is saved and
    retrieved.

    set: io_u->numberio=28489, off=0x5f2400
    queue: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    set: io_u->numberio=28574, off=0x5f2400
    iolog: overlap 6235136/512, 6235136/512
    queue: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    complete: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
    complete: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0

    retrieve: io_u->numberio=28574, off=0x5f2400
    queue: io_u 0x5b8039e1db40: off=0x5f2400,len=0x200,ddir=0,file=verify.0.0

    bad header numberio 28489, wanted 28574 at file verify.0.0 offset 6235136, length 512 (requested block: offset=6235136, length=512)

    Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>

[minwooim]

However your point stands - without a thing that can serialize ALL overlapping writes regardless of which concurrent job submitted them, you open yourself up to regions where you cannot know what value they have...

Indeed, so this PR serialize all the overlapping incoming submissions to guarantee that only a single write command can enter to the device at a time.