The semantics should be similar to that of k8s Job. And this will pave the work for the training-operator (https://github.com/kubeflow/training-operator/issues/1519)