NextFlow ~Error executing process > 'INTERPROSCAN:COMBINE:XREFS_LOCAL (10)' Caused by: Java heap space | Ubuntu 24.04

[desmodus1984]

Hi,
I ran InterproScan6 with 16 parallel processes/cores laptop/workstation with Linux, 16 cores/ 32 GB RAM.

I used the following code:

nextflow run ebi-pf-team/interproscan6     -profile apptainer --goterms     --input LW3.fasta --pathways     --datadir data --max-workers 16

It was running well and then I got this error:
I am adding the .nextflow.log file. It was confusing that it starts with .*

The command line error has this

Plus 7 more processes waiting for tasks…
WARN: Killing running tasks (9)
ERROR ~ Unexpected thread exception

 -- Check '.nextflow.log' file for details
ERROR ~ Error executing process > 'INTERPROSCAN:COMBINE:XREFS_LOCAL (10)'

Caused by:
  Java heap space


Source block:
  String outputFilePath = task.workDir.resolve('matches2xrefs.json')
  ProcessXrefs.run(matches_path.toString(), db_releases, add_goterms, add_pathways, panther_paint_dir, outputFilePath)

Work dir:
  /home/jpac1984/work/fb/c934e285e3389674eb8f4fbf8680ab

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

nextflow.zip

My OS is the @#$#@%# Ubuntu 24.04, and I need to run these before for nextflow to work:
Fix
sudo sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

Looking forward for your help to move on finishing the annotation. Just to remind, I am running InterproScan on workstation which has 32 GB of RAM, and 16 cores.

Hope you can help me fix this never-ending analysis - and no, I have access to a HPC but it uses old Linux libraries and many things that I try to run fail and I don't have external connectivity/Internet. I rely on my aforementioned laptop/workstation - where I have nextflow/apptatiner installed.

[Nickalus12]

To resolve the Java heap space error in the INTERPROSCAN:COMBINE:XREFS_LOCAL process, you need to increase the memory allocation for that specific task using a custom configuration file.

Create a file named custom.config in your current directory with the following content:

process {
    withName: 'INTERPROSCAN:COMBINE:XREFS_LOCAL' {
        memory = { 12.GB * task.attempt }
        cpus = 1
        errorStrategy = 'retry'
        maxRetries = 2
    }
}

Then, run your command with the -c flag to include this configuration:

nextflow run ebi-pf-team/interproscan6 \
    -profile apptainer \
    -c custom.config \
    --goterms \
    --input LW3.fasta \
    --pathways \
    --datadir data \
    --max-workers 16 \
    -resume

Why this is happening

The error Java heap space occurs when a process tries to load more data into memory than the JVM (Java Virtual Machine) is allowed to use. In the interproscan6 pipeline, the XREFS_LOCAL process merges cross-references and GO terms; for large input files (like your LW3.fasta), the JSON objects being processed can exceed the default memory settings.

How this fix works

• Targeted Override: The withName selector targets only the failing process, ensuring you don't waste RAM on smaller tasks.
• Dynamic Scaling: Using memory = { 12.GB * task.attempt } tells Nextflow to start with 12 GB, and if it fails again, try with 24 GB on the next attempt.
• Resource Management: Since your workstation has 32 GB of RAM and you are running 16 workers, you are likely oversubscribing your memory. While max-workers 16 uses all your cores, InterProScan processes are memory-intensive. If the error persists, consider lowering --max-workers to 8 to allow more RAM per individual process.

Tip

Since you are on Ubuntu 24.04 and already adjusted AppArmor settings, ensure your Apptainer cache directory has enough space, as InterProScan databases are quite large and can fill up /tmp quickly.

Does the process successfully move past the XREFS_LOCAL step with the increased memory allocation?