2024 01 29 webex

Agenda - 1/27/24

Discussion

Is the slice 1 ULFM targeting MPI 5.

Aurelian thinks for sessions we need to consider what happens if there's something that goes wrong with one or more processes before even getting into a communicator. Dan thinks there's a gap where with Sessions one doesn't start with communicators but process sets. Dan thinks we need to close this gap. In fail stop case, MPI_GROUP_FROM_SESSION_PSET we would not have holes. What we could use is a group of failed processes without having a communicator. Could at least support shrink if we had such a function. Aurelian suggests get failed set from a group operation.

Aurelian points out some challenges with implementation of the consensus algorithm. Assumption of no false positives - as is the case with ULFM now. Discussion of skipping presumed dead processes in a spanning tree. Assume no more failures while executing the consensus algorithm. Comm shrink currently makes use of pre-existing processes from the input communicator.

Briefly discuss whether session init/finalize and init again are things fixed? I.e. of one creates a new communicator based off of pset mpi://world will it be problem free?

We will cancel next week's sessions wg meeting and meet with the FT folks at 11 CST US on Monday 2/5/24.

Additional info provided by Dan

Hi all,

WORDS to capture our discussion on Monday 29th Jan.

The basic idea in the example code I gave is to detect process fail-stop faults, shrink the active communicator to exclude the failed processes, and then carry on.

The while loop captures the "carry on" part, only exit when we hit the break statement (or we suffer a process fail-stop fault). The additional code block captures the "shrink" part by creating a new communicator that excludes failed processes.

The MPI_Group_from_session_pset calls will all produce identical groups, even after processes have failed. This is a local procedure and there's no reason for it to return an error. We should guarantee that it never returns errors of class MPI_ERR_PROC_FAILED. The MPI_Session_get_proc_failed calls give a snapshot of the knowledge contained in the local detector. There is no communication or nonlocal dependence here. This is a (new) local procedure and there's no reason for it to return an error. We should guarantee that it never returns errors of class MPI_ERR_PROC_FAILED. The group manipulation procedures are existing MPI and will work exactly the same as they always have done. These are local procedures and there's no reason for them to return an error. We should guarantee that they never return errors of class MPI_ERR_PROC_FAILED.

The devils are always in the details. The details in this case are all in the implementation of MPI_Comm_create_from_group.

The MPI_Comm_create_from_group procedure must handle some difficult cases:

1. the potential for failed processes to exist in the group that is passed in by the calling MPI process

2. the potential for the group passed in to be different to the groups passed in by other MPI processes

3. the potential for the additional MPI processes to fail during the operation

4. failed before the operation If this MPI process attempts to communicate with a failed process, it's detector must eventually detect that, otherwise it is broken! This means that the failure of the other MPI process is discovered during the operation -- see point (3).

5. different groups at different processes There are several sub-cases here: a) procA's group includes procB but procB has failed (any time before procA communicates with it) b) procA's group contains procB but procB's does not include procA c) procA's group contains procB and procC but procB's group includes procA but does not include procC d) procA's group and procB's group are identical but one of them later discovers an (a|b|c) problem Current shrink implementations rely on shared knowledge of the survivors so each involved process can independently create the same spanning tree as all the other involved processes. This simplifies the design and results in a better worst-case performance scaling, at least asymptotically [ED: pls check] All of the above sub-cases can happen during a shrink operation, so intuitively this is no worse than that. We have two broad categories of approach here:

• any problem results in an error, return immediately without creating the output communicator
• any FT problem is dealt with internally; some kind of best-effort communicator is created We probably want a resilient algorithm that doesn’t have dreadful scaling.

3. failures during the operation We can just do an agreement at the end to catch failures that happen after the critical moment when other processes communicate with it.

Resilient algorithm: Start optimistic: Create a spanning tree from the group members you have been given in the call. Be eager to help: Always listen for incoming protocol messages related to communicator creation. Attempt communication with your direct children; if new failure detected, update the local group, update the local detector, fix the spanning tree (skip that child, but add that child's children as your direct children) if a child reports a different group (compare hashes), figure out the difference, update the local detector, fix the spanning tree (recreate it using only uncontacted undead processes) Attempt communication with your parent; if new failure detected, update the local group, update the local detector, fix the spanning tree (skip the parent, but add that parent's parent as your direct parent) if the parent reports a different group (compare hashes), figure out the difference, update the local detector, fix the spanning tree (recreate it using only uncontacted undead processes) Assume it all worked beautifully but execute MPI_Comm_agree to make certain. if new failure detected, update the local group, update the local detector, start again if the agreement succeeds without discovering new failures, we're done!

Best-case asymptotic scaling is O(log(p)), a spanning tree traversal of P processes. Worst-case O(sum_i=1,p-1(log(p-i)) < O(p log(p)), no more than p-1 tree traversals with a shrinking tree size.

Best wishes, Dan.