Multi Task Multi Stage Applications
A Bag-of-Tasks (BoT) represents a classic and the simplest example of task-level parallelism. We investigate the scenario of the multi-stage (M) BoT, where output data from task T(N,M) (i.e. the Nth task in stage M) is constrained to be assigned as input data for another T (N, M +1), but where T (N, M +1) might be arbitrarily located. In a stepwise approach we show that the pattern can be expressed as a GWENDIA dataflow workflow which in turn can be mapped to the Pilot-Data concept, which enables the transparent distribution of the dataflow workflow over multiple resources.
