Refactor dummy_region fixture as function by hannaseithe · Pull Request #3950 · digitalfabrik/integreat-cms

hannaseithe · 2025-10-17T09:50:56Z

Short description

Inside test_hix.py we use a dummy region with a dummy language in some tests. If this module is executed before the test test_no_changes_were_made_message inside test_poi_category_form_view.py the test fails because of an extra language in the db.

Proposed changes

refactor the dummy_region fixture as a simple function to be used inside the tests, this way the dummy_region and dummy_language are not committed to the db (only fixtures are persisted in the db)

Alternative Solution

keep the fixture and mark all tests that use the fixture as last, so that I are executed with tests that use transactional_db, where the db is flushed after every tests

Side effects

hopefully none, as no test should rely on the dummy region to be there

Faithfulness to issue description and design

There are no intended deviations from the issue and design.

How to test

Make sure that test_no_changes_were_made_message does not fail again on the CircleCi Pipeline in the coming weeks, with this kind of error-message

FAILED tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[CMS_TEAM] - assert 'Keine Änderungen vorgenommen' in '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'
 +  where '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <built-in method decode of bytes object at 0x557bab37e500>('utf-8')
 +    where <built-in method decode of bytes object at 0x557bab37e500> = b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'.decode
 +      where b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <TemplateResponse status_code=200, "text/html; charset=utf-8">.content
FAILED tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[ROOT] - assert 'Keine Änderungen vorgenommen' in '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'
 +  where '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <built-in method decode of bytes object at 0x557ba9d5ae00>('utf-8')
 +    where <built-in method decode of bytes object at 0x557ba9d5ae00> = b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'.decode
 +      where b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <TemplateResponse status_code=200, "text/html; charset=utf-8">.content
FAILED tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[SERVICE_TEAM] - assert 'Keine Änderungen vorgenommen' in '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'
 +  where '\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X-...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <built-in method decode of bytes object at 0x557baae71b40>('utf-8')
 +    where <built-in method decode of bytes object at 0x557baae71b40> = b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n'.decode
 +      where b'\n\n\n<!DOCTYPE html>\n\n<html lang="de">\n    <head>\n        <meta charset="utf-8" />\n        <meta http-equiv="X...      </div>\n        </div>\n    </form>\n\n        </div>\n    </main>\n\n        \n        \n    </body>\n</html>\n' = <TemplateResponse status_code=200, "text/html; charset=utf-8">.content

Resolved issues

Related to: #3949

Pull Request Review Guidelines

PeterNerlich · 2025-10-17T16:38:06Z

EDIT: I overlooked that a fixture is involved, which of course will not be part of the rollback after the test. It is still weird that my tests always passed, but the bug is definitely the fixture.

Did you observe this behaviour or is this just a conjecture?

I got a bit suspicious as the tests are only marked with django_db – if I remember correctly this should run the test in a transaction that is never committed but rolled back at the end of the test. Unfortunately, trying to prove this is very confusing.
But I just ran tools/test.sh -vv tests/cms/views/utils/test_hix.py tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[CMS_TEAM] a few times and quickly got it to run hix test first and last – without any test failing:

tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[CMS_TEAM] PASSED   [ 12%]
tests/cms/views/utils/test_hix.py::test_skip_textlab_call_if_html_is_empty PASSED                                   [ 25%]
tests/cms/views/utils/test_hix.py::test_contact_card_gets_filtered_out PASSED                                       [ 37%]
tests/cms/views/utils/test_hix.py::test_hix_rounding PASSED                                                         [ 50%]
tests/cms/views/utils/test_hix.py::test_versions_of_hix_page PASSED                                                 [ 62%]
tests/cms/views/utils/test_hix.py::test_disregard_pages_with_hix_ignore PASSED                                      [ 75%]
tests/cms/views/utils/test_hix.py::test_hix_values PASSED                                                           [ 87%]
tests/cms/views/utils/test_hix.py::test_disregard_archived_pages PASSED                                             [100%]

============================================= 8 passed, 4 warnings in 18.19s =============================================
✔ Tests successfully completed

tests/cms/views/utils/test_hix.py::test_hix_rounding PASSED                                                         [ 12%]
tests/cms/views/utils/test_hix.py::test_contact_card_gets_filtered_out PASSED                                       [ 25%]
tests/cms/views/utils/test_hix.py::test_skip_textlab_call_if_html_is_empty PASSED                                   [ 37%]
tests/cms/views/utils/test_hix.py::test_disregard_pages_with_hix_ignore PASSED                                      [ 50%]
tests/cms/views/utils/test_hix.py::test_hix_values PASSED                                                           [ 62%]
tests/cms/views/utils/test_hix.py::test_versions_of_hix_page PASSED                                                 [ 75%]
tests/cms/views/utils/test_hix.py::test_disregard_archived_pages PASSED                                             [ 87%]
tests/cms/views/poi_categories/test_poi_category_form_view.py::test_no_changes_were_made_message[CMS_TEAM] PASSED   [100%]

============================================= 8 passed, 4 warnings in 18.76s =============================================
✔ Tests successfully completed

Unless there is something wrong with this way of running it (e.g. we might suspect the verbosity or that it is not in CircleCI to introduce some side effect) or you actually observed the state of the database containing the dummy language, I think this disproves your statement that it still lingers after the test

hannaseithe · 2025-10-20T12:42:27Z

@PeterNerlich

I figured out why your tests were not failing in the second executing: The test test_hix_rounding is not using the dummy-region fixture
I have seen your edit, but just want to add, that you can observe the dummy-language being there in the log here:
https://app.circleci.com/pipelines/github/digitalfabrik/integreat-cms/16439/workflows/cabcaacd-09a3-4c03-a851-f35ff90d161e/jobs/226573?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Amharisch - language-id: 8 - slug:am
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Arabisch - language-id: 3 - slug:ar
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Deutsch - language-id: 1 - slug:de
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Deutsch (leicht) - language-id: 6 - slug:de-si
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language:  - language-id: 11 - slug:du
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Griechisch - language-id: 10 - slug:el
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Englisch - language-id: 2 - slug:en
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Spanisch - language-id: 5 - slug:es
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Farsi - language-id: 4 - slug:fa
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Hidden language - language-id: 7 - slug:hidden
DEBUG    test_poi_category_form_view:test_poi_category_form_view.py:248 Language: Ukrainisch - language-id: 9 - slug:uk

hannaseithe · 2025-10-20T12:47:40Z

Also I want to mention on another suggestion that @PeterNerlich brought up:

Instead of using a fixture and marking the test as last, we could simply create a function that does the creation of dummy region for us, that we call, wherever it is needed. In that case (if it is not a fixture), where the dummy region creation is part of the test itself, it will not be commited to the database and other tests should not fail if executed afterwards

hannaseithe · 2025-10-21T15:53:13Z

I have implemented Peter's suggestion now, as I agree that this is the cleaner solution, we should not use a fixture in such a situation

PeterNerlich

I figured out why your tests were not failing in the second executing: The test test_hix_rounding is not using the dummy-region fixture

I'm confused as to what this means, this doesn't explain to me why no test failed in either configuration. But making it not a fixture makes sense (rather, having it as a fix didn't make sense in the first place without something like #3872) and I just tried the same I did before, with no test failing. If you have any combination of tests that should result in a failure given the "right" order, I will try that too on develop and here!

hannaseithe · 2025-10-22T13:58:59Z

I'm confused as to what this means, this doesn't explain to me why no test failed in either configuration.

I am embarassed: Somehow I read your log output as "you only executed the one hix test test_hix_rounding ". Sometimes the brain does funny things. So yes, I still dont know why your tests did not fail.

hannaseithe requested review from JoeyStk, MizukiTemma, PeterNerlich, jarlhengstmengel and jonbulz October 17, 2025 09:51

hannaseithe mentioned this pull request Oct 17, 2025

Logs for debugging flaky test & rerun of failing container in specific order #3948

Closed

hannaseithe mentioned this pull request Oct 20, 2025

Investigate the Flakiness of tests on the CircleCI pipeline #3949

Open

hannaseithe changed the title ~~Mark hix tests as last~~ Refactor dummy_region fixture as function Oct 21, 2025

PeterNerlich approved these changes Oct 22, 2025

View reviewed changes

MizukiTemma approved these changes Oct 22, 2025

View reviewed changes

Mark hix tests as last

4969e15

JoeyStk force-pushed the flaky-test/test_no_changes_were_made_message branch from 0682207 to 4969e15 Compare October 22, 2025 16:09

JoeyStk merged commit 35332aa into develop Oct 22, 2025
4 of 5 checks passed

JoeyStk deleted the flaky-test/test_no_changes_were_made_message branch October 22, 2025 16:36

hannaseithe mentioned this pull request Oct 24, 2025

test_no_changes_were_made_message is flaky #3896

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor dummy_region fixture as function#3950

Refactor dummy_region fixture as function#3950
JoeyStk merged 1 commit intodevelopfrom
flaky-test/test_no_changes_were_made_message

hannaseithe commented Oct 17, 2025 •

edited

Loading

Uh oh!

PeterNerlich commented Oct 17, 2025 •

edited

Loading

Uh oh!

hannaseithe commented Oct 20, 2025 •

edited

Loading

Uh oh!

hannaseithe commented Oct 20, 2025 •

edited

Loading

Uh oh!

hannaseithe commented Oct 21, 2025

Uh oh!

PeterNerlich left a comment

Uh oh!

hannaseithe commented Oct 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

hannaseithe commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short description

Proposed changes

Alternative Solution

Side effects

Faithfulness to issue description and design

How to test

Resolved issues

Uh oh!

PeterNerlich commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hannaseithe commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hannaseithe commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hannaseithe commented Oct 21, 2025

Uh oh!

PeterNerlich left a comment

Choose a reason for hiding this comment

Uh oh!

hannaseithe commented Oct 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hannaseithe commented Oct 17, 2025 •

edited

Loading

PeterNerlich commented Oct 17, 2025 •

edited

Loading

hannaseithe commented Oct 20, 2025 •

edited

Loading

hannaseithe commented Oct 20, 2025 •

edited

Loading