[FLINK-38404][core] Fix JobMasterTest due to data race

[liuml07]

https://issues.apache.org/jira/browse/FLINK-38404

[flinkbot]

CI report:

• c5c4c12 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[liuml07]

I don't think e2e test failure is related

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16596ae]
...
Oct 09 05:08:29 [FAIL] 'Run kubernetes session test (default input)' failed after 1 minutes and 52 seconds! Test exited with exit code 1

[davidradl]

I had a quick look at this is. My concern is that the CREATED event might not be coming out - and this is a Flink bug. I would feel for confident if the test checks that the CREATED event is coming out and the code be arranged so that the race condition does not occur. WDYT?

[liuml07]

While this test cares more about the RUNNING → FAILING → FAILED events sequence when planned failures happen, I agree it's possible to hide a real problem in the Flink non-testing code. However, I do not find an easy fix that only touches the testing code to avoid the race condition. As the JobStatusChangeEvent is only for new job status (code), the initial CREATED is not expected to be captured here.

Reporting JobStatusChange event seems pretty new. @pnowojski may provide some ideas?

[liuml07]

If we'd like to change the Flink non-testing code, I have updated the PR to demonstrate the idea of notifying the jobStatus listeners at registration time. But I presume that's a larger code change, and just share for discussion. I'll look into the code more carefully.