Skip to content

Conversation

Akirathan
Copy link
Member

@Akirathan Akirathan commented Sep 17, 2025

Closes #13918

Pull Request Description

Update GraalVM from 24.0.1 to 25.0.0.

Important Notes

Highlights from GraalVM 25 release notes:

  • FFM API support on aarch64
  • Implemented Arena.ofShared
  • Experimental Native image tracing agent
  • Experimental -H:+JDWP flag.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

  • The documentation has been updated, if necessary.
  • Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
  • All code follows the
    Scala,
    Java,
    TypeScript,
    and
    Rust
    style guides. In case you are using a language not listed above, follow the Rust style guide.
  • Unit tests have been written where possible.
  • If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
    or the Snowflake database integration, a run of the Extra Tests has been scheduled.
    • If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

@Akirathan Akirathan self-assigned this Sep 17, 2025
@Akirathan Akirathan added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Sep 17, 2025
@Akirathan
Copy link
Member Author

NI build of engine-runner is successful. Its size is 533 MB. The previous size (for example built here) was 568 MB.

So just by upgrading to GraalVM 25.0.0, the NI size decreased by roughly 6%.

@Akirathan
Copy link
Member Author

Akirathan commented Sep 22, 2025

Failure in std-table/test: Mockito failure: "Java 25 not supported" at https://github.com/enso-org/enso/actions/runs/17914034971/job/50933411374?pr=14019#step:14:2018.

Fixed by bumping Mockito version in 6ac2d05

mergify bot pushed a commit that referenced this pull request Sep 23, 2025
#14019 is blocked by this - the current google java format version is failing because it tries to call remove method from JDK - see https://github.com/enso-org/enso/actions/runs/17916525266/job/50940077628?pr=14019#step:5:245

# Important Notes
The only notable change is in ff3fec5 . Every thing else is just automatic `sbt javafmtAll` formatting.
@Akirathan Akirathan marked this pull request as ready for review September 23, 2025 17:07
Object[] arguments,
EnsoHashMap warnings,
@Shared("loopNode") @Cached(value = "createLoopNode()", allowUncached = true) LoopNode loopNode) {
@Shared("loopNode") @Cached(value = "createLoopNode()", uncached = "createUncachedLoopNode()")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • It is surprising we need a change like this
  • isn't it regression?
  • Something to be reported to GraalVM guys?

@Akirathan
Copy link
Member Author

In Standard Library Native Tests (windows, amd64), 4 test suites (Visualization_Tests, AWS_Tests, Base_Tests, Generic_JDBC_Tests and Table_Tests) are failing on "Memory Access Violation" error. More specifically, the processes return 0xc0000005 exit code. The processes crash silently. There is no dump when this violation happens.

@Akirathan
Copy link
Member Author

Akirathan commented Sep 29, 2025

I asked @jdunkerley to reproduce. He was able to reproduce the problem on his Windows by running sbt runEngineDistribution --run test/Base_Tests:

[WARN] [2025-09-29T18:03:08+01:00] [org.enso.runner.JavaFinder] No appropriate JDK found in the distribution runtimes. Trying system-wide JDK.
[WARN] [2025-09-29T18:03:08+01:00] [org.enso.librarymanager.local.DefaultLocalLibraryProvider] Candidate library enso_dev.Base_Tests at [***\Base_Tests] may not be AOT ready! Use --jvm option when encoutering problems.
[WARN] [2025-09-29T18:03:08+01:00] [org.enso.librarymanager.local.DefaultLocalLibraryProvider] Candidate library enso_dev.Benchmarks at [***\Benchmarks] may not be AOT ready! Use --jvm option when encoutering problems.
[WARN] [2025-09-29T18:03:08+01:00] [org.enso.librarymanager.local.DefaultLocalLibraryProvider] Candidate library enso_dev.Generic_JDBC_Tests at [***\Generic_JDBC_Tests] may not be AOT ready! Use --jvm option when encoutering problems.
[WARN] [2025-09-29T18:03:08+01:00] [org.enso.librarymanager.local.DefaultLocalLibraryProvider] Candidate library enso_dev.Snowflake_Tests at [***\Snowflake_Tests] may not be AOT ready! Use --jvm option when encoutering problems.
[WARN] [2025-09-29T18:03:08+01:00] [org.enso.librarymanager.local.DefaultLocalLibraryProvider] Candidate library enso_dev.Table_Tests at [***\Table_Tests] may not be AOT ready! Use --jvm option when encoutering problems.
Sept 29, 2025 6:03:34 PM org.enso.interpreter.runtime.EnsoPolyglotJava createPolyglotJava
SEVERE: Using experimental OtherJvm support!
Picked up JAVA_TOOL_OPTIONS: -ea
Sept 29, 2025 6:03:34 PM org.enso.interpreter.runtime.TruffleCompilerContext getPackageOf
WARNING: null
java.nio.file.FileSystemException: C:\Repos\Enso\ide\test\Base_Tests\src\Data\Regression_Spec.enso: Invalid access to memory location
        at java.base@25/sun.nio.fs.WindowsLinkSupport.getRealPath(WindowsLinkSupport.java:280)
        at java.base@25/sun.nio.fs.WindowsPath.toRealPath(WindowsPath.java:907)
        at java.base@25/sun.nio.fs.WindowsPath$WindowsPathWithAttributes.toRealPath(WindowsPath.java:136)
        at org.graalvm.truffle/com.oracle.truffle.polyglot.FileSystems$NIOFileSystem.toRealPath(FileSystems.java:1057)
        at org.graalvm.truffle/com.oracle.truffle.polyglot.FileSystems$PreInitializeContextFileSystem.toRealPath(FileSystems.java:551)
        at org.graalvm.truffle/com.oracle.truffle.api.TruffleFile.getCanonicalFile(TruffleFile.java:538)
        at org.enso.interpreter.runtime.TruffleCompilerContext.getPackageOf(TruffleCompilerContext.java:1000)
        at org.enso.interpreter.runtime.EnsoContext.getPackageOf(EnsoContext.java:638)
        at org.enso.interpreter.caches.ModuleCache.getCacheRoots(ModuleCache.java:114)
        at org.enso.interpreter.caches.Cache.save(Cache.java:108)
        at org.enso.interpreter.runtime.TruffleCompilerContext.saveCache(TruffleCompilerContext.java:209)
        at org.enso.interpreter.runtime.TruffleCompilerContext.lambda$doSerializeModule$0(TruffleCompilerContext.java:460)
        at java.base@25/java.util.concurrent.FutureTask.run(FutureTask.java:328)
        at java.base@25/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:309)
        at java.base@25/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
        at java.base@25/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
        at java.base@25/java.lang.Thread.runWith(Thread.java:1487)
        at java.base@25/java.lang.Thread.run(Thread.java:1474)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:832)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:808)

The entire output is in message.txt

EDIT: There was no --no-ir-caches flag specified. But on the CI, all the tests are executed with --no-ir-caches.

@JaroslavTulach
Copy link
Member

at org.enso.interpreter.caches.ModuleCache.getCacheRoots(ModuleCache.java:114)
at org.enso.interpreter.caches.Cache.save(Cache.java:108)
at org.enso.interpreter.runtime.TruffleCompilerContext.saveCache(TruffleCompilerContext.java:209)

The stacktrace indicates a problem while storing .ir caches. That has an easy fix: don't store the caches. Whether or not that is sufficient to avoid

the processes return 0xc0000005 exit code. The processes crash silently

remains to be seen. However running without caches is the simplest check to make.

@JaroslavTulach
Copy link
Member

@Akirathan
Copy link
Member Author

Akirathan commented Sep 30, 2025

That has an easy fix: don't store the caches.

Caches are not stored on the CI. All the tests are executed with --no-ir-caches. I have verified locally, with:

Index: engine/runtime/src/main/java/org/enso/interpreter/caches/Cache.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/engine/runtime/src/main/java/org/enso/interpreter/caches/Cache.java b/engine/runtime/src/main/java/org/enso/interpreter/caches/Cache.java
--- a/engine/runtime/src/main/java/org/enso/interpreter/caches/Cache.java	(revision 3d9986b8e62fd416d681c0e7e6cfa27b71383321)
+++ b/engine/runtime/src/main/java/org/enso/interpreter/caches/Cache.java	(date 1759233097083)
@@ -104,6 +104,7 @@
    * @throws IOException if something goes wrong
    */
   public final TruffleFile save(T entry, EnsoContext context) throws IOException {
+    System.out.printf("[%s] Saving cache  %n", logName);
     TruffleLogger logger = context.getLogger(this.getClass());
     for (var root : spi.getCacheRoots(context)) {
       if (saveCacheTo(context, root, entry, logger)) {
@@ -176,6 +177,7 @@
    * @return the cached data if possible, and [[None]] if it could not load a valid cache
    */
   public final Optional<T> load(EnsoContext context) {
+    System.out.printf("[%s] Loading cache  %n", logName);
     var logger = context.getLogger(this.getClass());
     var collected = new ArrayList<IOException>();
     synchronized (LOCK) {

that when you run runEngineDistribution --no-ir-caches test/Base_Tests, no caches are saved. When you omit the --no-ir-caches flag, some caches are saved.

@JaroslavTulach
Copy link
Member

Caches are not stored on the CI.

I beg your pardon. What is being stored here?

java.nio.file.FileSystemException: C:\Repos\Enso\ide\test\Base_Tests\src\Data\Regression_Spec.enso: Invalid access to memory location
        at java.base@25/sun.nio.fs.WindowsLinkSupport.getRealPath(WindowsLinkSupport.java:280)
        at java.base@25/sun.nio.fs.WindowsPath.toRealPath(WindowsPath.java:907)
        at java.base@25/sun.nio.fs.WindowsPath$WindowsPathWithAttributes.toRealPath(WindowsPath.java:136)
        at org.graalvm.truffle/com.oracle.truffle.polyglot.FileSystems$NIOFileSystem.toRealPath(FileSystems.java:1057)
        at org.graalvm.truffle/com.oracle.truffle.polyglot.FileSystems$PreInitializeContextFileSystem.toRealPath(FileSystems.java:551)
        at org.graalvm.truffle/com.oracle.truffle.api.TruffleFile.getCanonicalFile(TruffleFile.java:538)
        at org.enso.interpreter.runtime.TruffleCompilerContext.getPackageOf(TruffleCompilerContext.java:1000)
        at org.enso.interpreter.runtime.EnsoContext.getPackageOf(EnsoContext.java:638)
        at org.enso.interpreter.caches.ModuleCache.getCacheRoots(ModuleCache.java:114)
        at org.enso.interpreter.caches.Cache.save(Cache.java:108)

I'd bet that Cache.save in ModuleCache object is trying to save caches. Are you denying this evidence?

@Akirathan
Copy link
Member Author

I beg your pardon. What is being stored here?

This output was copied from James. And for the first time, James executed the command without --no-ir-caches. See the EDIT section in that comment.

@JaroslavTulach
Copy link
Member

Are you denying this evidence?

EDIT: There was no --no-ir-caches flag specified

OK, now I get it.

@Akirathan
Copy link
Member Author

Akirathan commented Oct 1, 2025

Reaching out to GraalVM slack for help - https://graalvm.slack.com/archives/CN9KSFB40/p1759317558077949

@Akirathan
Copy link
Member Author

To reproduce this error on local Windows machine:

env ENSO_LAUNCHER=native,fast,-ls sbt runEngineDistribution --no-ir-caches --run test/Base_Tests

@Akirathan
Copy link
Member Author

Akirathan commented Oct 1, 2025

With e1251ac, I have confirmed that the "Memory Access Violation" error happens even without saving or loading any caches. There is no "Saving cache", neither "Loading cache" output in https://github.com/enso-org/enso/actions/runs/18104093889/job/51517127762?pr=14019

@Akirathan
Copy link
Member Author

Akirathan commented Oct 1, 2025

With 8cdb45c, Standard Library native Windows test will run only Base_Tests with --log-level TRACE.

EDIT: The output in https://github.com/enso-org/enso/actions/runs/18160937708/job/51694612696 is useless. There is just a lot of output from the compiler. It is still not clear from the error is encountered. Let's also print the currently running test in 8262f97

@Akirathan
Copy link
Member Author

Received suggestions from the GraalVM team how to debug the issue:

  1. Build NI with -H:+PreserveFramePointer -H:-DeleteLocalSymbols -H:-RemoveUnusedSymbols -H:+SourceLevelDebug and attach Windows debugger. https://graalvm.slack.com/archives/CN9KSFB40/p1759320670277689?thread_ts=1759317558.077949&cid=CN9KSFB40
  2. Try different optimization options (-O0, -Ob, ...). https://graalvm.slack.com/archives/CN9KSFB40/p1759320610282109?thread_ts=1759317558.077949&cid=CN9KSFB40
  3. Capture minidump. procdump -i C:\Dumps to globally hook crashes like that and open the generated dump in Windows debugger. https://graalvm.slack.com/archives/CN9KSFB40/p1759327285722709?thread_ts=1759317558.077949&cid=CN9KSFB40

CI uses Ignore_Progress_Reporter.
@Akirathan
Copy link
Member Author

Akirathan commented Oct 3, 2025

How to reproduce the Memory Access Violation error?

After debugging session with @jdunkerley, we observed that:

  • On the CI, the failing test is in Data_Spec.
  • @jdunkerley reproduced the error when evaluating "should allow creating a directory" in File_Spec.
    • Executed with env ENSO_LAUNCHER=native,-ls,fast sbt runEngineDistribution --no-ir-caches --run test/Base_Tests/src/System/File_Spec.enso should.allow.creating.a.directory
    • After bisecting with IO.println statements, the line that causes the error is
      f.delete_if_exists recursive=True
    • So recursive deletion of directory is failing.
    • When recursive=False, no error is encountered.
    • When executing the code of the test in a separate file, no error is encountered.
    • The coredump produced as suggested on GraalVM Slack is uploaded in Google Drive.
      • Unfortunately, Windows does not support debugging symbols, so the coredump is not very useful.
      • Debugging symbols tried with building NI with -H:+PreserveFramePointer -H:-DeleteLocalSymbols -H:-RemoveUnusedSymbols -H:+SourceLevelDebug.

The snippets that @jdunkerley confirmed to work properly without error:

main =
    file = File.new "C:/Temp/Hello.csv"
    "Testing".write file
main =
    file = File.new "C:/Temp/A/B/C/D"
    file.create_directory
    file2 = File.new "C:/Temp/A"
    file2.delete recursive=True
main =
    file = File.new "C:/Temp" / "good_dir"
    file.delete_if_exists
    file.create_directory
    IO.println file.exists

    file2 = file / "bar" / "baz"
    file2.create_directory
    IO.println file2.exists

    file2.delete_if_exists
    IO.println file2.exists
    
    file2.parent.delete_if_exists
    IO.println file2.parent.exists

    file.delete_if_exists recursive=True
    IO.println file.exists
GitHub
Enso Analytics is a self-service data prep and analysis platform designed for data teams. - Update to GraalVM 25 · 5d6779f

@JaroslavTulach
Copy link
Member

JaroslavTulach commented Oct 3, 2025

With:

enso> git diff
diff --git a/engine/runtime/src/main/java/org/enso/interpreter/runtime/data/EnsoFile.java b/engine/runtime/src/main/java/org/enso/interpreter/runtime/data/EnsoFile.java
index 5c449f716f..aa43c25199 100644
--- a/engine/runtime/src/main/java/org/enso/interpreter/runtime/data/EnsoFile.java
+++ b/engine/runtime/src/main/java/org/enso/interpreter/runtime/data/EnsoFile.java
@@ -697,17 +697,27 @@ public final class EnsoFile extends BuiltinObject {
     if (recursive && file.truffleFile.isDirectory(LinkOption.NOFOLLOW_LINKS)) {
       deleteRecursively(file.truffleFile);
     } else {
+      System.err.println("simple delete: " + file.truffleFile);
       file.truffleFile.delete();
     }
   }
 
+  @TruffleBoundary
   private static void deleteRecursively(TruffleFile file) throws IOException {
+    System.err.println("deleteRecursively: " + file);
     if (file.isDirectory(LinkOption.NOFOLLOW_LINKS)) {
       for (TruffleFile child : file.list()) {
         deleteRecursively(child);
       }
     }
-    file.delete();
+    System.err.println("do delete: " + file);
+    try {
+      file.delete();
+    } catch (Throwable t) {
+      t.printStackTrace();
+      throw t;
+    }
+    System.err.println("  - OK");
   }
 
   @Builtin.Method(name = "copy_builtin", description = "Copy this file to a target destination")
enso$ ENSO_LAUNCHER=native,-ls,fast sbt
sbt:enso> runEngineDistribution --no-ir-caches --run test/Base_Tests/src/System/File_Spec.enso should.allow.creating.a.directory

 (1 / 1) File Operations: should allow creating a directory
simple delete: S:\windows\enso\test\Base_Tests\data\good_dir
simple delete: S:\windows\enso\test\Base_Tests\data\good_dir\bar\baz
simple delete: S:\windows\enso\test\Base_Tests\data\good_dir\bar
deleteRecursively: S:\windows\enso\test\Base_Tests\data\good_dir
[warn] built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\bin\enso.exe finished with exit code -1073741819

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: Clean build required CI runners will be cleaned before and after this PR is built.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to GraalVM 25 LTS
4 participants