Motivation
We have quite a few tests that flake on the circleci pipeline. The investigation is challenging, as the local test setup (with its own flakiness) cannot help in reproducing the flaky tests from circleci. Therefore we have to depend on debugging through logging there.
It might also be useful to re-consider the circleci-cli tool. But the problem here is that locally the tool cannot handle multiple docker containers, as we use during our test job on circleci.
The test_no_changes_were_made is the first one we started investigating. The #3948 PR is our first attempt at log-debugging the failing tests. And will hopefully lead is to a better understanding where the instability actually results from.
One suspicion is that somehow the test isolation fails and data from one test leaks into another. But why and how that happens is still not clear yet.
Problem statement
Research through log-debugging (or other means) what causes the flakiness in our tests on the circleci pipeline
Additional Context
This also relates to #3872, which aims at gaining better test isolation through the cloning of the db instead of relying on our session/function load_test_data setup (which has been suspected to be the culprit here)
Fixed Tests
Motivation
We have quite a few tests that flake on the circleci pipeline. The investigation is challenging, as the local test setup (with its own flakiness) cannot help in reproducing the flaky tests from circleci. Therefore we have to depend on debugging through logging there.
It might also be useful to re-consider the
circleci-clitool. But the problem here is that locally the tool cannot handle multiple docker containers, as we use during ourtestjob on circleci.The
test_no_changes_were_madeis the first one we started investigating. The #3948 PR is our first attempt at log-debugging the failing tests. And will hopefully lead is to a better understanding where the instability actually results from.One suspicion is that somehow the test isolation fails and data from one test leaks into another. But why and how that happens is still not clear yet.
Problem statement
Research through log-debugging (or other means) what causes the flakiness in our tests on the circleci pipeline
Additional Context
This also relates to #3872, which aims at gaining better test isolation through the cloning of the db instead of relying on our session/function load_test_data setup (which has been suspected to be the culprit here)
Fixed Tests
test_no_changes_were_made_message: fixed by Refactor dummy_region fixture as function #3950 (?)test_duplicate_pages - django.core.exceptions.ValidationError: ['Slug kann nicht aus dem Titel abgeleitet werden.']: fixed by Refactor dummy_region fixture as function #3950 (?)