Nix Integration after reworking P2P

[zhongjiechen]

#624 removes the RDMA impl in uccl-collective from P2P and changes some behaviors. It may no longer be compatible with uccl-backend in Nixl, requiring additional integration effort.

[praveingk]

@zhongjiechen I am trying to refactor the NIXL integration to be one-sided without needing advertise execution. Can prepare_fifo_metadata be called from the initiator ?

Because I see mr_array is being used to add padding to the fifo_item:

copyRKeysFromMRArrayToBytes(mhandle->mr_array,
static_cast<char*>(remote_mem_info.padding),
sizeof(remote_mem_info.padding));

[praveingk]

I am exploring if this fifo_item can be calculated during memory registration, and be shared using NIXL's existing meta exchange. Then on the initiator side, the fifo_item should be stored for each MR.

[zhongjiechen]

The main purpose of this should be to store the Rkeys of different NICs. We need to store them in advance when the NIXL agent exchanges metadata/establishes a connection, so that the initiator can use them directly.

[praveingk]

@zhongjiechen I was able to craft the Fifo item and perform read/write. Now, moving on the supporting notifications.
Should this support be added to rdma_endpoint, since it handles the oob_client and oob_server?

[zhongjiechen]

@zhongjiechen I was able to craft the Fifo item and perform read/write. Now, moving on the supporting notifications. Should this support be added to rdma_endpoint, since it handles the oob_client and oob_server?

Is it worthwhile to implement it using RDMA rather than OOB for lower latency, if it's on the critical path?

[praveingk]

No, it's ok to OOB, since it doesn't require low latency. Additionally, this can be common across multiple transport (tcp, tcpx).

[zhongjiechen]

Got it! Probably add the logic to epoll_client.h and epoll_server.h i guess?