WIP: MPI shared memory

[hmenke]

No description provided.

[hmenke]

Open questions from meeting Tue Feb 25, 11:00 EST

• Generalize AddressSpace concept and move away from integer arithmetic for combine functionality.
• Refactor handle_heap (and possibly other handles) to allow blk_shm_t.
• Should mpi_shm_allocator be stateful, so that is fulfills the Allocator concept, or should we have a separate concept/interface?
• Make mpi::shared_communicator a static member of mpi_shm_allocator such that mpi_shm_allocator can fulfill the Allocator concept.
• handle_heap (and possibly other handles) need to somehow expose the userdata field in case of a blk_fat_t.
• handle_borrowed needs same API as handle_heap to be able to access userdata.

Idea for API:

blk_t {
  char * ptr;
  size_t s;
  window* win;
}

handle_heap {
  blk_t blk;

  requires ...
  window *win() { return blk.win; }
}

shared_array {
  handle_heap sto;
  void fence() { sto.win->fence(); }
}

[hmenke]

Currently handle_borrowed will forget what was the allocator of the Handle that is has borrowed from.

The handle_heap has a defaulted Allocator A template argument

  template <typename T, Allocator A = mallocator<>>
  struct handle_heap {

but handle_borrowed does not have an Allocator A template argument and simply wraps a handle_heap with the default mallocator<>. Furthermore, the constructor from a Handle H will only assign _parent if H is the default handle_heap<T0>.

  template <typename T, AddressSpace AdrSp = Host>
  struct handle_borrowed {
    private:
    using T0 = std::remove_const_t<T>;
    handle_heap<T0> const *_parent = nullptr;

    /* ... omitted ... */

    template <Handle H>
      requires(address_space == H::address_space and (std::is_const_v<value_type> or !std::is_const_v<typename H::value_type>)
               and std::is_same_v<const value_type, const typename H::value_type>)
    handle_borrowed(H const &h, long offset = 0) noexcept : _data(h.data() + offset) {
      if constexpr (std::is_same_v<H, handle_heap<T0>>) _parent = &h;
    }

This is a bit problematic, because it means that creating a handle_borrowed from a non-default handle_heap will either fail or simply forget that original handle altogether.

Thus getting the parent() of a handle_borrowed is not possible if it was borrowed from a handle that is not handle_heap with the default allocator.