`test_free_threading.test_monitoring` flaky (occasional reference leak)

[colesbury]

Bug report

When running ./python -E -m test --timeout=2700 -R3:3 -u-cpu -j40 test_free_threading.test_monitoring -F, I get a failure every ~50-100 test runs:

0:00:36 load avg: 122.12 [ 55/1] test_free_threading.test_monitoring failed (reference leak)
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XX. 243
test_free_threading.test_monitoring leaked [2, 4, 3] memory blocks, sum=9

I think the issue is that we rely on Thread.is_alive() instead of joining the threads. I think there are still some objects that may not be freed until the thread is joined (like the handle):

cpython/Lib/test/test_free_threading/test_monitoring.py

Lines 65 to 77 in cb59eae

	while True:
	any_alive = False
	for t in threads:
	if t.is_alive():
	any_alive = True
	break
	if not any_alive:
	break
	self.during_threads()
	self.after_test()

Linked PRs

• gh-136692:Improve test cleanup and thread handling in monitoring #136896

[colesbury]

There's a second source of flakiness: sys.monitoring lazily allocates monitoring data the first time a code object is run under instrumentation. The test_free_threading.test_monitoring flips turns on and off monitoring during the test run. It's possible that the test case finishes with monitoring still enabled and tearDown turns it off.

cpython/Lib/test/test_free_threading/test_monitoring.py

Lines 125 to 142 in 180b3eb

	def tearDown(self):
	monitoring.set_events(self.tool_id, 0)
	super().tearDown()
	def callback(self, *args):
	self.called = True
	def after_test(self):
	self.assertTrue(self.called)
	def during_threads(self):
	if self.set:
	monitoring.set_events(
	self.tool_id, monitoring.events.CALL | monitoring.events.LINE
	)
	else:
	monitoring.set_events(self.tool_id, 0)
	self.set = not self.set

This can lead to a false reported reference leak. If you are unlucky, the first three warmup tests will finish with monitoring disabled. If a subsequent run finishes with monitoring still enabled, then parts of unittest get instrumented before tearDown() is called leading to new lazily allocated monitoring data.

Adding a monitoring.set_events(self.tool_id, 0) to after_test() helps, but there are similar (but rare) issues with other code paths getting non-deterministically executed under monitoring in some runs but not others.

[colesbury]

@gaogaotiantian, I'd like to get your take on this:

We have some flakiness in refleak tests that use threads and sys.monitoring. Basically, there's some non-determinism in terms of which code objects get run under monitoring so sometimes _co_monitoring and other monitoring-related fields get initialized after the "warmup" runs, and get (mis)reported as refleaks.

To avoid flakiness, I'm thinking about clearing all unused monitoring data in either sys._clear_internal_caches or test.libregrtest.utils.clear_caches, at least in the free threading build. What do you think about this?