Tests randomly fail when running entire rspec test suite (and don't when run alone)

[reinvanimschoot]

• I have tried upgrading by running bundle update vite_ruby.
• I have read the troubleshooting section before opening an issue.

Description 📖

I am trying to migrate a Rails/Inertia application from webpacker to Vite.

So far, everything seems to work except for the tests. When I run the whole test suite tests seem to randomly fail that don't fail when I run them separately. I tested this in the webpacker environment where I did not have this issue.

I tried setting autoBuild: false and adding ViteRuby.commands.build to my rails_helper.rb but that only made other tests fail for some reason.

I'm not sure what could be the problem here but I would love to get this to work. Any help would be greatly appreciated.

Reproduction 📖

I cannot really provide a full fledged reproduction but I can tell you the failing tests all use multiple Capybara sessions ( to test interaction).

Run bin/rake vite:info and provide the output:

vite_ruby: 3.3.4
vite_rails: 3.0.15
rails: 6.1.5
node: v18.12.0
npm: 8.19.2
yarn: 1.22.19
pnpm: 
ruby: ruby 3.2.0 (2022-12-25 revision a528908271) [arm64-darwin22]

installed packages:
brightbook@ /Users/rein/workspace/brightbook
├─┬ @vitejs/[email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
├── [email protected]
└─┬ [email protected] invalid: "^0.31.0" from node_modules/vitest-dom
  ├─┬ [email protected]
  │ └── [email protected]
  └── [email protected]

Logs 📜

If not providing a reproduction:

Output

Run DEBUG=vite-plugin-ruby:* bin/vite dev or DEBUG=vite-plugin-ruby:* bin/vite build and provide the output:

  vite-plugin-ruby:config {
  vite-plugin-ruby:config   base: '/vite-dev/',
  vite-plugin-ruby:config   build: {
  vite-plugin-ruby:config     emptyOutDir: true,
  vite-plugin-ruby:config     sourcemap: false,
  vite-plugin-ruby:config     rollupOptions: { input: [Object], output: [Object] },
  vite-plugin-ruby:config     assetsDir: 'assets',
  vite-plugin-ruby:config     manifest: true,
  vite-plugin-ruby:config     outDir: '../../public/vite-dev'
  vite-plugin-ruby:config   },
  vite-plugin-ruby:config   envDir: '/Users/rein/workspace/brightbook',
  vite-plugin-ruby:config   root: '/Users/rein/workspace/brightbook/app/javascript',
  vite-plugin-ruby:config   server: {
  vite-plugin-ruby:config     fs: { allow: [Array], strict: true },
  vite-plugin-ruby:config     host: 'localhost',
  vite-plugin-ruby:config     https: false,
  vite-plugin-ruby:config     port: 3036,
  vite-plugin-ruby:config     strictPort: true,
  vite-plugin-ruby:config     hmr: { clientPort: 3036 }
  vite-plugin-ruby:config   },
  vite-plugin-ruby:config   entrypoints: {
  vite-plugin-ruby:config     'entrypoints/active_admin.js': '/Users/rein/workspace/brightbook/app/javascript/entrypoints/active_admin.js',
  vite-plugin-ruby:config     'entrypoints/active_admin.scss': '/Users/rein/workspace/brightbook/app/javascript/entrypoints/active_admin.scss',
  vite-plugin-ruby:config     'entrypoints/application.tsx': '/Users/rein/workspace/brightbook/app/javascript/entrypoints/application.tsx',
  vite-plugin-ruby:config     'entrypoints/print.tsx': '/Users/rein/workspace/brightbook/app/javascript/entrypoints/print.tsx',
  vite-plugin-ruby:config     'assets/fonts/Roboto-Black.ttf': '/Users/rein/workspace/brightbook/app/javascript/assets/fonts/Roboto-Black.ttf',
  vite-plugin-ruby:config     'assets/fonts/Roboto-Bold.ttf': '/Users/rein/workspace/brightbook/app/javascript/assets/fonts/Roboto-Bold.ttf',
  vite-plugin-ruby:config     'assets/fonts/Roboto-Light.ttf': '/Users/rein/workspace/brightbook/app/javascript/assets/fonts/Roboto-Light.ttf',
  vite-plugin-ruby:config     'assets/fonts/Roboto-Medium.ttf': '/Users/rein/workspace/brightbook/app/javascript/assets/fonts/Roboto-Medium.ttf',
  vite-plugin-ruby:config     'assets/images/editor.gif': '/Users/rein/workspace/brightbook/app/javascript/assets/images/editor.gif',
  vite-plugin-ruby:config     'assets/images/online.gif': '/Users/rein/workspace/brightbook/app/javascript/assets/images/online.gif',
  vite-plugin-ruby:config     'assets/images/print.gif': '/Users/rein/workspace/brightbook/app/javascript/assets/images/print.gif'
  vite-plugin-ruby:config   }
  vite-plugin-ruby:config } +0ms

Screenshots 📷

Provide console or browser screenshots of the problem.

[ElMassimo]

Are you using a parallel runner? See this discussion:

• Flaky test due to missing assets #357 (comment)

and the Integration Tests in the CI section of the guide.

adding ViteRuby.commands.build to my rails_helper.rb but that only made other tests fail for some reason.

Please provide more information, what is the reason?

Try running RAILS_ENV="test" bin/rake assets:precompile before running tests to verify if it's a synchronization problem.

[reinvanimschoot]

Thanks for the fast reply!

I tried setting autoBuild: falseand I removed ViteRuby.commands.buildin favor of RAILS_ENV="test" bin/rake assets:precompilebut I'm still having the same issues.

The only tests that fail seem to be the ones that have two capybara sessions running. So I'm not sure where the problem could be lying.

[reinvanimschoot]

This is an example test that fails.

The interesting thing is, when I remove the first small test, so the file only has one test, the test passes when running be rspec spec/features/custom-question/interaction_spec.rb. When I uncomment the first test, so the file has two tests, the previous command fails BUT running the test with a line number still passes, like so: rspec spec/features/custom-question/interaction_spec.rb: 44.

I'm honestly out of ideas.

# frozen_string_literal: true

require "rails_helper"

feature "custom questions", :js, type: :feature do
  let(:book) { create(:book) }

  let(:teacher_license) { create_claimed_license!(role: :teacher, book: book) }
  let(:student_license) { create_claimed_license!(book: book) }

  let(:class_group) { create(:class_group, name: "klasgroep", teacher_user_book: teacher_license.user_book) }

  let(:section) { create(:section, :with_content, content: "definition", book: book) }

  let(:section_page) { Page::Section.new(section) }

  before do
    s(:teacher) { login teacher_license.user }
    s(:student) { login student_license.user }

    class_group.user_books << student_license.user_book

    s(:teacher) { section_page.visit }
    s(:student) { section_page.visit }
  end

  it "cannot be asked when no class group is selected by the teacher" do
    s(:teacher) { expect(section_page).to have_no_class_group_selected }
    s(:teacher) { expect(section_page.ask_question_button).to be_disabled }
  end

  context "when teacher has class group selected" do
    before do
      s(:teacher) { section_page.select_class_group(class_group) }
    end

    it "can create interaction without any question input" do
      s(:teacher) { click_on("Stel een vraag") }
      s(:teacher) { click_on("Deel") }

      s(:teacher) { expect(section_page).to have_content("Annuleer Vraag") }

      s(:student) { expect(section_page).to have_answer_question_button }

      s(:student) { section_page.answer_question_button.click }

      s(:student) { expect(section_page).to have_content("Verstuur") }

      s(:student) { fill_in("answer-input", with: "Leerling antwoord") }
      s(:student) { click_on("Verstuur") }

      s(:student) { expect(section_page).to have_content("Je antwoord is verzonden!") }

      s(:teacher) { expect(section_page).to have_content("Leerling antwoord") }

      s(:teacher) { click_on("Annuleer Vraag") }
      s(:teacher) { expect(section_page).not_to have_content("Annuleer Vraag") }

      s(:student) { expect(section_page).not_to have_answer_question_button }
    end
  end

  def session(name, &block)
    Capybara.using_session(name) do
      block.call
    end
  end

  alias_method :s, :session
end

[ElMassimo]

The fact that running a single test works as expected suggests that this is a problem when cleaning up resources, like a hanging Capybara session.

Please provide the test failure messages—so far it doesn't seem like this is related with Vite Ruby.

[reinvanimschoot]

So I got the tests to work but not quite.

When I do
RAILS_ENV="test" VITE_RUBY_AUTO_BUILD="false" bin/rake assets:precompile
and run
bundle exec rspec,
the test suite always passes all tests.

However, when I add something like this in my rails_helper.rb (as seen in the linked issue)

  config.before :suite do
    puts "\e[31mPrecompiling assets for feature specs \e[0m"
    ViteRuby.commands.build
    puts "\e[32m-> Finished! precompiling assets for system specs \e[0m"
  end

there are some tests that fail, where the screenshot is a white page.

So I'm assuming there still is a timing/sync issue somewhere? For CI, this isn't really a problem, since we could just the precompile as a step before the tests, but when adding or updating tests, you don't want to have to manually precompile every time.

[ElMassimo]

Are you using a parallel runner?

[reinvanimschoot]

No, at least not locally. I haven't tried the tests in CI.

[ElMassimo]

Seems unusual.

the screenshot is a white page.

I'd suggest outputting the HTML of that page, or the server response (if any).

Moving to Discussions, as it's likely a problem with the integration tests setup.

[reinvanimschoot]

So I was able to fix this, it turns out the problem lies at the side of Inertia. When I downgraded from 1.x.x to 0.x.x all my problems disappeard. Very strange and I'm not sure where the combination of Inertia/Vite breaks down but at least I got it working.