Skip to content

CI system tests start with different state than local system tests #1570

@trichoplax

Description

@trichoplax

Describe the bug
I made a new pull request #1569 to try to narrow down what problem I had introduced that was causing CI system test failures in #1548. However, before I replicated any of the changes I had made, the fresh new pull request already showed a difference between CI and local tests. I've included screenshots below that show that basic_user sees 7 posts in the Main category when system tests are run locally, but sees 0 posts when run on the CI.

To Reproduce
Steps to reproduce the behavior:

I added the following test to a new branch from develop:

require 'application_system_test_case'

class CIFailuresTest < ApplicationSystemTestCase
  test 'temporary test to analyse CI failures what does basic user see in main' do
    category = categories(:main)
    log_in :basic_user
    visit category_path(category)
    assert_text 'Nonexistent to force screenshot'
  end
end

Making a pull request then triggered the CI tests to run, and I compared the forced screenshot from the CI with the forced screenshot from running the system tests locally with rails test:system.

Expected behavior
The test on CI and locally should start with the same state, being based on the same fixtures. CI and local tests cannot be expected to have consistent output if they do not start with consistent state.

Screenshots
Locally:

Main category page showing 7 posts

On CI:

Main category page showing 0 posts

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: testingChanges to testing infrastructurecomplexity: unassessedNeeds further developer investigation before complexity/feasibility can be determined.priority: high

    Type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions