Summary
Take a look at this CR comment. It's correctly pointing out that the job will be marked as SUCCESS when the task finishes, even though at that point only the queueing has happened. I started looking into this (this is how I came across fix #1062), but looks like the Job status is generally tied to the state of the celery task that runs it.
We can add something like this:
class Job:
...
def update_from_task(task, status, retval, ...):
# default implementation does what
# ami/jobs/tasks:update_job_status() does
# But then MLJob can override to handle the async case.
Related Issues
Depends on #987.