Replace pwrite with asynchronous write

[prasad-joshi]

ASD uses synchronous reads/writes for IOs on disk/file devices.
During startup ASD creates a thread per storage backend. The thread
consumes a single ssd_write_buf from queue and issues a synchronous
write to disk. As a result, only a single write IO can be
outstanding at a time.

The size of the Q from which writer thread consumes writes is limited.
If the writer thread is not able to flush at the rate of client, ASD
start issuing OVERLOAD errors.

This is a first attempt to issue asynchronous writes to device, raising
the pull request just to get feedback. The work is not complete,
however I would like to get early review.

I have written a benchmark tool, which write 4096 sized objects with
various IO depths (from 1 to 128). It always fails with OVERLOAD error.
Besides, while the benchmark is running, IOSTAT reports only single
outstanding IO to disk.

The overload error is gone with the modified code and the IO
Request Q depth reported by IOSTAT is continuously more than 1.

The code change includes

1. During startup for each SSD device
   • create a additional cf_poll, register a eventfd for AsyncIO
   • create a separate eventfd for flush_write_q an shadow_write_q.
     Both the eventfds are registered with same poll.
2. Removed the thread which serves shadow write q. Create a single
   thread per SSD for all writes
3. signal eventfd, when a ssd_write_buf is pushed into any q.
4. The writer thread
   • reads as many ssd_write_buf as possible, fills IOCB, and
     issues multiple writes in a single system call.
   • The writer thread also handle the write completions

Signed-off-by: Prasad Joshi [email protected]

[kportertx]

Sorry for not seeing this sooner, I think github has stopped sending me notifications for pull requests on this repo.

With these changes, what is the expected behavior of the a disk that is falling behind (assume it will continue getting further and further behind)?