Extraction memory usage on HPC

[bmichanderson]

Hi Edgardo,

I'm trying to run Captus on a supercomputer and have run into some odd behaviour during step 3 (extraction). Despite providing 128 cores and 230GB of RAM, the step immediately fails due to an OOM error. When I run with --debug this doesn't happen, so it seems to relate to the parallelisation(?). I have 76 assemblies using skimming data (assembled with Captus), and I get the error when running 32 concurrent and 16 concurrent (c. 7 and 14 GB RAM per extraction).

Here's a portion of the log file:

           Captus version: v1.4.1
                  Command: /opt/conda/envs/captus_env/bin/captus extract --captus_assemblies_dir 02_assemblies --out ./03_extractions/ --nuc_refs /scratch/pawsey0220/anderson/Adenanthos/captus/HERB_targets_ref_prot.fasta --nuc_min_score 0.2 --nuc_min_identity 80 --nuc_min_coverage 20 --ram 235 --threads 128 --concurrent 16
                       OS: Linux-5.14.21-150500.55.83_13.0.62-cray_shasta_c-x86_64-with-glibc2.36
                     Host: nid002821
                 Max. RAM: 235.0GB (out of 250.3GB)
             Max. Threads: 128 (out of 256)

             Dependencies:
                Scipio   : v1.4.1 OK
                ├─BioPerl: v1.7.8 OK
                └─YAML   : v1.30 OK
                BLAT     : v37x1 OK
                MMseqs2  : not used
                MAFFT    : not used

         Python libraries:
                    numpy: v2.2.5 OK
                   pandas: v2.2.3 OK
                   plotly: v6.0.1 OK

    Captus assemblies dir: /scratch/pawsey0220/anderson/Adenanthos/captus/02_assemblies
                           VALID Captus assemblies directory with 76 samples and 76 'assembly.fasta' files

   Total assemblies found: 76

         Output directory: /scratch/pawsey0220/anderson/Adenanthos/captus/03_extractions
                           Output directory successfully created



Protein and DNA Markers Extraction from FASTA assemblies (2025-05-13 16:39:48)
    Now Captus will search within your FASTA assemblies for the set of
reference proteins provided by '--nuc_refs', '--ptd_refs', and/or '--mit_refs'
and extract the sequences both in nucleotide and translated to aminoacid using
the specified '--nuc_transtable', '--ptd_transtable' and '--mit_transtable'
respectively.
    Additionally, miscellaneous DNA references provided with '--dna_refs' will
also be extracted.

Verifying protein reference sets...

          PROTEIN OPTIONS:
  Max. loci for Scipio x2: 2000
  Predict dubious introns: False
   Concurrent extractions: 16
       RAM per extraction: 14.7GB
   Threads per extraction: 8

         Nuclear proteins:
                reference: /scratch/pawsey0220/anderson/Adenanthos/captus/HERB_targets_ref_prot.faa
           reference info: 284 loci, 284 sequences (loci names found, detected a single sequence per locus)
        translation table: 1
                min_score: 0.2
             min_identity: 80.0
             min_coverage: 20.0
          depth_tolerance: 2.0 (min = 10^(log(depth of the contig with best hit in locus) / 2.0))

      Plastidial proteins:
                reference: not used

   Mitochondrial proteins:
                reference: not used


MISCELLANEOUS DNA OPTIONS:
                reference: not used


           OUTPUT OPTIONS:
 Ignore depth of coverage: False
 Disable contig stitching: False
            Max. paralogs: -1 (Keep all paralogs)
        Paralog tolerance: 5
 Max. separate loci files: 0 (Do not write separate loci files per sample)
          Overwrite files: False
           Keep all files: False

       Samples to process: 76
   Extractions to process: 76 protein and 0 nucleotide

         Nuclear proteins: /scratch/pawsey0220/anderson/Adenanthos/captus/03_extractions/[Sample_name]__captus-ext/01_coding_NUC
     Annotated assemblies: /scratch/pawsey0220/anderson/Adenanthos/captus/03_extractions/[Sample_name]__captus-ext/06_assembly_annotated
                           (These output directories will be created for each sample)

 Paths to references used: /scratch/pawsey0220/anderson/Adenanthos/captus/03_extractions/captus-extract_refs.json

Extracting protein-coding markers with Scipio:
  0%|                                            | 0/76 [00:00<?, ?extraction/s]slurmstepd: error: Detected 1 oom_kill event in StepId=24882035.0. Some of the step tasks have been OOM Killed.
srun: error: nid002821: task 0: Out Of Memory
srun: Terminating StepId=24882035.0

Any ideas? Is it possible the Scipio step is requesting more memory than available?

Cheers,
Ben

[bmichanderson]

as an update, this also hits an error when I set it to use all threads for each extraction (so, concurrent = 1)

   Concurrent extractions: 1
       RAM per extraction: 235.0GB
   Threads per extraction: 128

[bmichanderson]

I may have discovered (partly) why the runs are failing. It seems like the RAM argument is not explicitly used by the Scipio commands outside of adjusting threads for BLAT(?), so I adjusted the threads argument to the call (cut it in half) and it is now running. It appears that there are issues in my set up if there are fewer than 3 GB RAM per thread. This may be a problem others encounter when there are a large number of CPUs and not enough memory.

             Max. RAM: 235.0GB (out of 250.3GB)
             Max. Threads: 64 (out of 256)
...
   Concurrent extractions: 16
       RAM per extraction: 14.7GB
   Threads per extraction: 4

[edgardomortiz]

Hi Ben,

It is great you could solve the issue. I was going to explain the BLAT behavior but you deduced it correctly already. When deciding default values for parallelization I take into account the size and number of sequences in your target file, however the size of the assemblies is also very important (and harder to predict because of variability across samples) because that is what BLAT loads into RAM for the search. So to give you an example, let's say your computer has 32 cores and 128GB of RAM, if you force --concurrent 4 --threads 32 then you will run 4*8 instances of BLAT, if your assemblies are larger than 4GB then you exceed the 128GB of RAM. So your solution is what I was going to recommend because it seems that you have many/all big assemblies? If just a couple are big and the rest not, sometimes increasing concurrency helps because you can dedicate more RAM in less threads to the big ones and the rest of the load is distributed across smaller assemblies...

Just out of curiosity, what happens when you let Captus decide the number of concurrent searches? does it fail too?

Edgardo

[edgardomortiz]

Maybe we should keep this issue opened in case other people run into the same problem. By the way, those Proteaceae look really cool!

[bmichanderson]

Hi Edgardo,

Thanks for the quick response. Yes, if I set --concurrent auto it also fails. And yes, my assemblies are skimming data and large. I guess the weird behaviour is the memory error when I set --concurrent 1. If Captus is only loading one assembly into memory, there shouldn't be an issue, but I think somehow the memory per thread is still a bottleneck (i.e. lots of memory but way more threads). Somehow the trigger to parallelise is leading to the mem/thread problem.
Glad you enjoyed the plants :). This project is looking at historical samples in the group to sort out whether a taxon is legitimate.

Ben

[edgardomortiz]

Hi Ben,

Captus runs BLAT in parallel. For example, if you have 32 cores and run --concurrent 1 Captus will split the target file in 32 pieces and run 32 instances of BLAT, loading the entire assembly into RAM up to 32 times. I tried the other way around, splitting the assembly in 32 pieces and searching the entire target file on each but the results are inconsistent.

Since it seems this run is RAM-limited, the solution is what you described, to limit the number of cores available to Captus. Ideally, take the size of your RAM and divide it by the mean size of your assemblies and provide the integer as --threads letting Captus to decide the concurrency. I will try to automate this in a future version, could I ask you then to test it?

Your project sounds exactly to what I did with Cucumis, paper pending unfortunately...

Edgardo

[bmichanderson]

ah, I see. Well, I will have to see how it goes with future jobs for setting up the threads. If I've got an active dataset when the next version comes out, I'm happy to give it a crack.

[edgardomortiz]

Hi again Ben,

I added the functionality in the latest version (1.4.6), Captus reserves RAM based on the size of your assemblies and splits CPUs automatically based on that. I also fixed a small bug related to TAPER affecting the alignment step.

Thanks,

Edgardo

[bmichanderson]

Hey Edgardo, thanks for that!
I am now running the new version with the same assemblies and it seems to be functioning as expected.

             Max. RAM: 235.0GB (out of 250.3GB)
             Max. Threads: 128 (out of 256)
...
   Concurrent extractions: 16
   RAM per extraction: 14.7GB
   Threads per extraction: 4

Previously it would have set threads to 8 per extraction.
This is with concurrent="auto".

Cheers,
Ben

[edgardomortiz]

Excellent! thanks for reporting!