Dynamically changing EmptyQueueBehavior from Sleep to anything else can leave threads permanently sleeping

[RichieSams]

In multiple places in the code, we switch based on the empty queue behavior and do different logic for each.

Most prominantly, when FiberStart function fails to find a task or fiber to run, it will either loop, yield, or sleep. If behavior is set to Sleep, in AddTask(s) and when waiting tasks are ready, we then notify sleeping threads to wake.

const EmptyQueueBehavior behavior = m_emptyQueueBehavior.load(std::memory_order_relaxed);
if (behavior == EmptyQueueBehavior::Sleep) {
    // Find a thread that is sleeping and wake it
    for (size_t i = 0; i < m_numThreads; ++i) {
        std::unique_lock<std::mutex> lock(m_tls[i].FailedQueuePopLock);
        if (m_tls[i].FailedQueuePopAttempts >= kFailedPopAttemptsHeuristic) {
            m_tls[i].FailedQueuePopAttempts = 0;
            m_tls[i].FailedQueuePopCV.notify_one();

            break;
        }
    }
}

However, if the user changes the behavior mid run, we can end up with a thread permanently sleeping:

1. Worker thread A goes to sleep since there are no tasks
2. User changes behavior from Sleep to Yield
3. User code calls AddTasks()
4. AddTasks() doesn't wake thread A because behavior is set to Yield now.