Zinc incremental compilation behaviour is different when executed in the single jvm vs two consecutive runs

[ov7a]

steps

Please see the reproducer

• https://github.com/ov7a/zinc-example/blob/main/src/main/scala/Main.scala
  and its execution:
• https://github.com/ov7a/zinc-example/actions/runs/11699629356/job/32581996234

In short, it does the following:

1. creates a mixed java/scala project
2. compiles java sources with javac and scala sources with zinc
3. changes the java source and recompiles it
4. recompiles the scala source with zinc

If all these steps are executed in a single sbt run, zinc does not see the updated java class properly. If 1-2 and 3-4 are made in two consecutive runs, it works as expected.

I can observe this behavior with zinc 1.10.4, 1.10.0, 1.9.6.

problem

I suspect it has something to do with the internal caches of zinc. However, I haven't found a proper way to clear them.

expectation

I expect the behavior to be the same: the state should be preserved in compileAnalysis, and all the necessary data should be there. Apparently, that's not the case.

I expect the same output to be produced if the same inputs are used, independent of the environment.

notes

Somebody may suspect that it depends on the loaded jars, but I get the same issue with the hardcoded, explicit paths to pre-downloaded jars.

The reproducer is a very rough approximation of how Gradle (8.10) works.

I stumbled across this issue while investigating

• Scala Zinc Incremental compiler regression as of version 1.10.1 gradle/gradle#30898

The only relevant change in 1.10.1 is fixed stamps for empty files/directories, but I can't reproduce Gradle's issue in isolation. If anyone has ideas about what can be an (implicit) input, it would be appreciated.

[eed3si9n]

If you want Java sources to participate in the incremental compilation, then you might need to use Zinc to compile Java so we can analyze them. I didn't read the repro project in detail, but https://github.com/ov7a/zinc-example/blob/4fadc653f6d37f64ec9dd829e5d8b87bf3b14184/src/main/scala/Main.scala#L94-L103 looks suspicious, if it's literally just calling javac.

[ov7a]

If you want Java sources to participate in the incremental compilation, then you might need to use Zinc to compile Java so we can analyze them.

Is there a way to use zinc only for scala incremental compilation? In other words, is there an API to provide info about java classes changes? Since zinc is supposed to be used in build tools, I would expect it to be agnostic to that (Imagining fictional scenario with groovy or someNewJVMLangugage compilation).

if it's literally just calling javac.

Yes, it does. I used the plain call for simplicity.

Even if the scenario is wrong, I don't understand why, despite providing the same inputs to zinc compiler, I (should) get different results depending on the method of invocation.

[eed3si9n]

Is there a way to use zinc only for scala incremental compilation?

Zinc can compile against JAR files that was compiled from pure Java, so you would probably have to pass information in that way. The mixed compilation is not just a Zinc thing, but it's a feature of Scala compiler to be able to interoperate Scala and Java entities, so Zinc provides an incremental compilation on top of it.

Even if the scenario is wrong, I don't understand why, despite providing the same inputs to zinc compiler, I (should) get different results depending on the method of invocation.

Incremental compilation works by tracking artifacts outputs and language level dependencies. So if the relevant symbols are not tracked, we can't really expect anything to behave correctly. On the other hand, if you could reproduce your result with sbt, then maybe there's a caching bug or whatever?

[ov7a]

Zinc can compile against JAR files that was compiled from pure Java, so you would probably have to pass information in that way.

Understood, thanks for clarification.

Incremental compilation works by tracking artifacts outputs and language level dependencies. So if the relevant symbols are not tracked, we can't really expect anything to behave correctly.

Sorry, I don't get that. I understand that the scenario in the reproducer may be incorrect, but this does not explain the inconsistency between the results. I expect the result to be the same (even if it's a wrong result) when I provide the same inputs to Zinc. Can you at least give some pointers on where to look for implicit inputs?

On the other hand, if you could reproduce your result with sbt, then maybe there's a caching bug or whatever?

Can you clarify, please? I think I narrowed it down to Zinc already. In the reproducer, "sbt run" is used for convenience, but I can reproduce the behavior without sbt (running in IDE). I can update the reproducer to run it with plain java if needed.

[eed3si9n]

Can you at least give some pointers on where to look for implicit inputs?

Programs are sequence of p⇒q logic, where the precondition p needs to hold for the result to be ok. Otherwise, it's undefined or unsound. I think, Zinc makes the assumption that all *.class files were created by Zinc, so no raw javac or kotlin or Clojure etc. If anything appeared to work correctly, I think it just happened to work. Specifically in your first run, second run case, Zinc didn't have useful information so it let scalac run in the second run probably?

Can you clarify, please?

What I'm saying is the if you can reproduce the first run, second run type of behavior using sbt, which is a reference usage of Zinc, then that might point to some bug in Zinc is all I'm saying.

[ov7a]

Workaround: -Dxsbt.skip.cp.lookup=true.

I noticed that when both runs happen in the same JVM, lookup.classpath becomes null after IncrementalCommon.isLibraryModified here. I didn't debug deeper, though, but it feels weird that immutable val is changed - at least, that's what the debugger shows. Maybe it's because of ParVector here.