Generating and Attaching of Scaladocs are broken with Scala3

[chr78rm]

The production of Scaladoc APIs are broken when using Scala3, e.g. with Scala 3.1.2:

<profile>
    <id>scala3-failure</id>
    <properties>
        <scala-maven-plugin.version>4.6.1</scala-maven-plugin.version>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>${scala-maven-plugin.version}</version>
                <executions>
                    <execution>
                        <id>Scalac</id>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                            <goal>add-source</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-deprecation</arg>
                                <arg>-release:17</arg>
                                <arg>-encoding</arg>
                                <arg>utf-8</arg>
                            </args>
                        </configuration>
                    </execution>
                    <execution>
                        <id>Scaladoc-jar</id>
                        <goals>
                            <goal>doc-jar</goal>
                        </goals>
                        <phase>package</phase>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala3-library_3</artifactId>
            <version>3.1.2</version>
        </dependency>
    </dependencies>
</profile>

This gives

$ mvn clean install -P scala3-failure
...
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ scaladoc-jar-test ---
[INFO] Building jar: C:\Users\Developer\Git\Git-2.36.0-64-bit\projects\scaladoc-jar-test\target\scaladoc-jar-test-0.0.1-sources.jar
[INFO]
[INFO] >>> scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) > generate-sources @ scaladoc-jar-test >>>
[INFO]
[INFO] --- scala-maven-plugin:4.6.1:add-source (Scalac) @ scaladoc-jar-test ---
[INFO]
[INFO] <<< scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) < generate-sources @ scaladoc-jar-test <<<
[INFO]
[INFO]
[INFO] --- scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) @ scaladoc-jar-test ---
bad option '-doc-format:html' was ignored
bad option '-doc-title' was ignored
source file not found: scaladoc-jar-test 0.0.1 API
2 warnings found
1 error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

The issue can be reproduced with the build profile scala3-failure of the test project. The test project can be cloned by

git clone https://github.com/chr78rm/scaladoc-jar-test.git

The mentioned profile uses the currently promoted version (4.6.1) of the plugin and tries to attach scaladocs as depicted on the project site. I have investigated this a bit further. One reason why this cannot work is the wrong qualified class name given by the the method apidocMainClassName(). The correct qualified class name would be dotty.tools.scaladoc.Main. Using Scala3 this class is even separated from the scala compiler, that is an additional dependency (scaladoc_3) is needed to resolve this class. Unfortunately a potential fix in accordance with the outline of the given program structure of the plugin runs into classloader issues. scaladoc_3 uses the java.lang.ClassLoader.getResourceAsStream(String) method but it seems that the invocations prepared by e.g. JavaMainCallerByFork don't provide a class loader when calling getClassLoader(). I am not an expert in classloader issues but it might well be that this issue is inevitable when using the currently provided invocations within the scala_maven_executions package.

Nevertheless, I have prepared a possible fix within the forked repo

https://github.com/chr78rm/scala-maven-plugin

of your plugin. Checkout the branch scaladoc3. The prepared fix uses the ProcessBuilder of the JDK to invoke a forked process and reflects as far as possible the given outline of the plugin code structure. The mentioned classloader issue can be triggered by checking out the branch classloader-issue. Both mentioned branches of the fork produce an artifact of the plugin with version 4.6.2-EXPERIMENTAL. This version is used by the other build profiles of the test project.

[chr78rm]

I have updated the test project to use the currently promoted version 4.6.2 within the build profile scala3-failure. Your latest commits have been integrated into the fork of your plugin. The branches scaladoc3 and classloader-issue are producing the version 4.6.3-EXPERIMENTAL now. Again, these version is referenced by the other build profiles of the reproducer.

[slandelle]

@chr78rm Thanks for reporting.
Sadly, I can't review your fork:

• lots of intermediate/trial-and-error/back-and-forth commits
• not rebased on upstream's HEAD
• is there an integration test proving your fix?

I'll only be able to review if you can provide a PR that fixes the above points (all the more as I personally use neither Scala 3 nor Scaladoc).
Regards

[chr78rm]

Thank you for your answer. I didn't want to put too much effort into this beforehand.

Please don't read too much into some of the commits on the branches. I needed to research your code and hence I had included some tracing statements at some time. After I understood the inner workings of the plugin well enough I had removed them again. Besides, I had updated the master branch of the forked repo to your latest commit (eac2c9b). After that, I integrated your latest commits including the fix for issue #600 into the scaladoc3 branch by working on the output of git diff master..scaladoc3 --name-only. That is to say the files listed by git diff master..scaladoc3 --name-only contain the relevant changes for the possible fix at this instant. All your commits up to (eac2c9b) are included in scaladoc3 now.

$ git diff master..scaladoc3 --name-only
pom.xml
src/main/java/scala_maven/ScalaDocMojo.java
src/main/java/scala_maven/ScalaMojoSupport.java
src/main/java/scala_maven_dependency/ArtifactIds.java
src/main/java/scala_maven_dependency/ArtifactIds4Scala2.java
src/main/java/scala_maven_dependency/ArtifactIds4Scala3.java
src/main/java/scala_maven_dependency/Context.java
src/main/java/scala_maven_dependency/Context4ScalaHome.java
src/main/java/scala_maven_dependency/Context4ScalaRemote.java
src/main/java/scala_maven_executions/ScalaDoc3Caller.java

I have attached a file with the full diffs to this comment but you might want to use an IDE for a better visual experience.

Additionally, I will replay the relevant diffs into a new branch rooted at your latest commit. I haven't looked into how you are organizing your integration tests yet. I'm using the mentioned test project as reference at this stage. E.g. the test project contains a build profile scala2 which can be used to verify that everything works in Scala2 as before. Another build profile works with Scala3.

Of course I can provide a pull request for this in due course.

diff-scaladoc3-master.txt

[hohonuuli]

Is there a way to disable arguments that are used to execute the scaladoc? Under the hood the Scala-maven-plugin scaladoc execution is running a command like:

 /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java -Xbootclasspath/a:/Users/brian/.m2/repository/com/google/protobuf/protobuf-java/3.7.0/protobuf-java-3.7.0.jar:[...] -classpath /Users/brian/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.8.0/scala-maven-plugin-4.8.0.jar scala_maven_executions.MainWithArgsInFile dotty.tools.dotc.Main /private/var/folders/d7/4ps3zfnx08n8gxq0wc9ydddw0000gq/T/scala-maven-2306955749493420450.args

The problem I seem to hit is that the temp file (e.g. /private/var/folders/d7/4ps3zfnx08n8gxq0wc9ydddw0000gq/T/scala-maven-2306955749493420450.args) contains the following lines:

-doc-format:html
-doc-title
"scommons 0.0.4_3 API"

These flags are are no longer appropriate for scaladoc 3 and when included produce the following:

bad option '-doc-format:html' was ignored
bad option '-doc-title' was ignored
source file not found: scommons 0.0.4_3 API
2 warnings found
1 error found

When I manually edit those out of the args file and rerun the command above, it works fine. So being able to strip those out of the args file might resolve this issue.

[hohonuuli]

Code that produces the bad flag is here

[mnd999]

The flags are old and deprecated but they shouldn't break anything according to scala/scala3#11907. The problem seems to be the newline that gets generated between -doc-title and the module name itself. If the offending lines in the file become as follows, we get warnings but it doesn't fail anymore:

-doc-format:html
-doc-title "scommons 0.0.4_3 API"

This would seem like a reasonably quick workaround to do.

[jam01]

Hi all, just hit this in 4.9.1. Is there any user-level workarounds?

---

edit: from what I've found so far, it seems in Scala 3 the plugin should be using a different class dotty.tools.scaladoc.Main? This is what gradle is using also

---

edit2: getting a little further with:

configuration:
  scaladocClassName: dotty.tools.scaladoc.Main
  dependencies:
    - groupId: org.scala-lang
      artifactId: scaladoc_3
      version: 3.3.3

forgive the yaml syntax (trying out polyglot)

I get the following with displayCmd

... scala_maven_executions.MainWithArgsInFile dotty.tools.scaladoc.Main /tmp/scala-maven-8998361167489940263.args
bad option '-doc-format:html' was ignored
scaladoc supports only .tasty and .jar files, following files will be ignored: ...
java.lang.NullPointerException: Cannot invoke "java.lang.ClassLoader.getResourceAsStream(String)" because the return value of "java.lang.Class.getClassLoader()" is null
	at dotty.tools.scaladoc.renderers.Resources.renderResource(Resources.scala:557)
	at dotty.tools.scaladoc.renderers.Resources.renderResource$(Resources.scala:20)
	at dotty.tools.scaladoc.renderers.Renderer.renderResource(Renderer.scala:20)
	at dotty.tools.scaladoc.renderers.HtmlRenderer.renderResources$$anonfun$1(HtmlRenderer.scala:74)
	at scala.collection.immutable.List.flatMap(List.scala:293)
	at scala.collection.immutable.List.flatMap(List.scala:79)
	at dotty.tools.scaladoc.renderers.HtmlRenderer.renderResources(HtmlRenderer.scala:74)
	at dotty.tools.scaladoc.renderers.HtmlRenderer.render(HtmlRenderer.scala:48)
	at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:239)
	at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:69)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.Option.map(Option.scala:242)
	at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:73)
	at dotty.tools.scaladoc.Main.run(Main.scala:8)
	at dotty.tools.scaladoc.Main$.main(Main.scala:14)
	at dotty.tools.scaladoc.Main.main(Main.scala)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at scala_maven_executions.MainHelper.runMain(MainHelper.java:156)
	at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:30)

---

edit3: adding args: ['-nobootcp'] prevents the error and the goal succeeds, but there's no actual docs in the -javadoc.jar. I'm confident it's because all the files passed listed are .scala which are not supported...

---

edit4: working configuration for scala 3.3.3

- id: attach-javadocs
  goals: ['doc-jar'] # on package
  configuration:
    args: ['-nobootcp'] # avoid NPE from getClass.getClassLoader
    scaladocClassName: dotty.tools.scaladoc.Main # scala3 class
    sourceDir: '${project.build.outputDirectory}' # use compiled classes
    includes: ['**/*.tasty'] # only tasty
    dependencies:
      - groupId: org.scala-lang
        artifactId: scaladoc_3
        version: 3.3.3

[chr78rm]

I can confirm that the configuration above does the trick for now. But it seems unreliable at best because it depends on an unsupported option (-nobootcp). Indeed I am getting the output: "bad option '-nobootcp' was ignored", but it wasn't actually ignored since removing the option causes the ClassLoader bug again.

The Compiler Options Lookup Table notes this option as not yet available in Scala 3.3.x and you cannot find this option within the ScalaDoc Settings at all. The same applies when listing the available options via scalac -help or scaladoc -help on the command line.

I had submitted a pull request (#611) two years ago which would have fixed this issue but the maintainers had chosen to ignore it.

The ScalaDoc generation additionally suffers from this bug at present: Scaladoc generates broken local links.

[slandelle]

I had submitted a pull request (#611) two years ago which would have fixed this issue but the maintainers had chosen to ignore it.

@chr78rm This is both not correct and offending.
I requested that you clean up your PR (squash commit, remove extra dependency to one of your personal projects) and rebase it as it was conflicting with the main branch before I could review.
This has not been done yet.

Please understand that I have some professional interest in maven + Scala 2 but have none for maven + Scala 3 and never will.
Still, I'm willing to spend some of my personal time to help solve other people's problems. But only to a certain extend.
Help me help you.

[chr78rm]

Please, calm down @slandelle . I have simply stated the fact that you chose to ignore the pull request, speak: to not integrate it - for whatever reasonable or unreasonable grounds you might have had. I really don't know how this comes as offending. Bye, Bye.

[slandelle]

I haven't chosen to ignore it. I've requested you to perform some changes before I could review it, which you haven't.

[johanhaleby]

I'm having the very same problem as we speak. I can't release my open-source project (rest-assured) because of this :( Will try the workarounds by @jam01 .