`CounterStrike.counter_example` is called at compile time, and can cause ElixirLS to crash

[Adam-McDevitt-Prima]

Hello

I've encountered an issue that makes development with propcheck difficult for me when using ElixirLS.

As far as I can tell, the language server attempts to call Counterstrike.counter_example at compile time, but the target process cannot be found. This causes ElixirLS to crash.

Error Message

[Info  - 16:17:09] Terminating Elixir.ElixirLS.LanguageServer.ExUnitTestTracer: exited in: GenServer.call(PropCheck.CounterStrike, {:counter_example, {PropCheck.TreeTest, :property_delete, []}}, 5000)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
[Error - 16:17:09] GenServer ElixirLS.LanguageServer.ExUnitTestTracer terminating
** (stop) exited in: GenServer.call(PropCheck.CounterStrike, {:counter_example, {PropCheck.TreeTest, :property_delete, []}}, 5000)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir 1.18.3) lib/gen_server.ex:1121: GenServer.call/3
    (propcheck 1.5.1-dev) lib/properties.ex:171: PropCheck.Properties.tag_property/1
    test/tree_test.exs:13: (module)
    (elixir 1.18.3) src/elixir_compiler.erl:77: :elixir_compiler.dispatch/4
    (elixir 1.18.3) src/elixir_compiler.erl:52: :elixir_compiler.compile/4
    (elixir 1.18.3) src/elixir_module.erl:454: :elixir_module.eval_form/7
    (elixir 1.18.3) src/elixir_module.erl:156: :elixir_module.compile/7
    (elixir 1.18.3) src/elixir_compiler.erl:38: :elixir_compiler.maybe_fast_compile/2
    (elixir 1.18.3) src/elixir_lexical.erl:13: :elixir_lexical.run/3
    (elixir 1.18.3) src/elixir_compiler.erl:17: :elixir_compiler.quoted/3
    (elixir 1.18.3) lib/module/parallel_checker.ex:120: Module.ParallelChecker.verify/1
    (language_server 0.27.2) lib/language_server/ex_unit_test_tracer.ex:84: anonymous fn/1 in ElixirLS.LanguageServer.ExUnitTestTracer.handle_call/3
    (kernel 10.2.7) global.erl:699: :global.trans/4
    (language_server 0.27.2) lib/language_server/ex_unit_test_tracer.ex:77: ElixirLS.LanguageServer.ExUnitTestTracer.handle_call/3
    (stdlib 6.2.2) gen_server.erl:2381: :gen_server.try_handle_call/4
    (stdlib 6.2.2) gen_server.erl:2410: :gen_server.handle_msg/6

To Reproduce

Tools

• elixir: 1.18.3 (installed via asdf)
• erlang: 27.3.4 (installed via asdf)
• VSCode: 1.100.0
• ElixirLS VSCode extension: 0.27.2

Steps

• Open propcheck repository in VSCode (I'd expect the same issue is present in other editors when using the same language server)
• Navigate to test/tree_test.exs
• Make some edits inside of a forall block (I've found that just deleting a Tree.member function call and then adding it back is sufficient.)
• ElixirLS should crash, with an error like above

Note that I first encounted this issue in a repo that was using propcheck as a dependency, so it is not exclusive to the propcheck repository. It's just convenient that it can be reproduced in the propcheck repo itself.

My Investigation

I think that the root cause is from CounterStrike.counter_example being called in Properties.tag_property. From my understanding, the tags are computed at compile time and so CounterStrike.counter_example must be called at compile time.

If I replace:

  def tag_property({m, f, a}) do
    mfa = {m, String.to_atom("property_#{f}"), a}

    case CounterStrike.counter_example(mfa) do
      {:ok, _} ->
        # Logger.debug "Found failing property #{inspect mfa}"
        true

      _ ->
        false
    end
  end

with:

  def tag_property({m, f, a}) do
    false
  end

then the issue stops, and I can edit propcheck tests without errors. (Although this is of course not a proper fix, as it breaks the counter example tagging functionality.)

I'm fairly new to Elixir and I don't have a lot of experience with macros/metaprogramming, so it's possible I've misunderstood something here.

Related Issues

Somebody reported what seems like the same issue on the ElixirLS repo.

[lukaszsamson]

I'll add some context. ElixirLS tries to compile all *_test.exs files via Code.compile_file in https://github.com/elixir-lsp/elixir-ls/blob/master/apps/language_server/lib/language_server/ex_unit_test_tracer.ex#L84 with a custom tracer that reads ex_unit_tests module attribute. This tracer extracts info about ExUnit test cases and uses that to populate TestUI. If the test file executes non trivial code expecting dependencies during compile time the compilation crashes and no tests from that file can be shown on UI.