If there is an error before the job submission, the error is logged in the cron log but the job is retried until the job/build is cancelled, leading to buildkite waiting forever.
We should treat any error before job submission the same way we treat job submission failures: raise an error in an error reporting buildkite job . This can be done in the except at the bottom of bin/poll.py