extensive RAM usage

[anna-chiaraB]

Hello,

I’ve been trying to run patchwork on a test dataset, but the job consistently aborts due to excessive memory usage.

For a dataset of ~113–119 million reads (trimmed and filtered, not assembled), the job uses up to ~450 GB RAM (12 CPUs) before being aborted. This exceeds the available ressources on our system. Is there any workaround to this problem?

      singularity run \
    --bind /media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork:/media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork \
    /opt/bioinformatics/containers/patchwork.sif \
    --contigs "/media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork/26613_1.fq.gz" \
            "/media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork/26613_2.fq.gz" \
    --reference "/media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork/Cylisticus_convexus-29744.longest_orfs_fixed_header.pep" \
    --output-dir /media/inter/abart/Projekte/Isopoda/GenomeSkimming/output/Patchwork/26613 \
    --threads 12

Thank you for your help!
Best, Anna

[fethalen]

Hi Anna,

That is indeed a lot of memory, so it is no surprise that this exceeds your that of your available resources. Depending on where the problem happens, the amount of memory used depends on the number of threads requested. It could be that lowering the number of threads used (safest bet would be setting --threads to 1) would circumvent this issue, but I'm not sure.

Would you mind sharing a log (stdout + stderr) of your job, so that I could see at which step it fails?

Best,
Felicia

[anna-chiaraB]

Hi Felicia,

I reran the job with explicit stdout/stderr logging
[285023] signal (15): Terminated in expression starting at none:1

The diamon_blastx.log and diamond_makedb.log in the output directory look normal and complete.

Thank you!
Best, Anna

[fethalen]

Hi Anna, I'm missing the stdout that would typically look something like this:

$ julia --project=. src/Patchwork.jl --contigs test/07673.fna  --reference test/07673_Alitta_succinea.faa 
P A T C H W O R K v0.7.2
- Developers: Felicia Sandberg, Clara G. Köhne    - Cite      : 10.1093/gbe/evad227
- Contact   : <f.sandberg@cardio-care.ch>         - DIAMOND v.: 2.1.8
- Docs      : github.com/fethalen/patchwork/wiki  - Threads   : 12
──────────────────────────────────────────────────────────────────────────
INPUT NEEDED: found output from a previous run, overwrite old files? (y/n):y
[ Info: Sequence data file format detected: FASTA
[ Info: Building DIAMOND database
[ Info: Aligning query sequences against reference database
[ Info: Merging overlapping hits
[ Info: Trimming alignments
──────────────────────────────────────────────────────────────────────────
# query files        : 1
# query sequences    : 1
# query nucleotides  : 1048
# reference sequences: 1
# alignments out     : 1 (100.0%)
┌────────────────┬───────┬───────┬────────┬───────┐
│       variable │  mean │   min │ median │   max │
├────────────────┼───────┼───────┼────────┼───────┤
│  reference_len │ 354.0 │   354 │  354.0 │   354 │
│      query_len │  18.0 │    18 │   18.0 │    18 │
│        regions │   1.0 │     1 │    1.0 │     1 │
│        contigs │   1.0 │     1 │    1.0 │     1 │
│        matches │  16.0 │    16 │   16.0 │    16 │
│     mismatches │   2.0 │     2 │    2.0 │     2 │
│      deletions │ 336.0 │   336 │  336.0 │   336 │
│ query_coverage │  5.08 │  5.08 │   5.08 │  5.08 │
│       identity │ 88.89 │ 88.89 │  88.89 │ 88.89 │
└────────────────┴───────┴───────┴────────┴───────┘
Query coverage in markers:
            ┌                                        ┐ 
   0.01-20% ┤■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 1   
     21-40% ┤ 0                                        
     41-60% ┤ 0                                        
     61-80% ┤ 0                                        
    81-100% ┤ 0                                        
            └                                        ┘ 
Percent identity in markers:
             ┌                                        ┐ 
     missing ┤ 0                                        
       ≤ 30% ┤ 0                                        
    > 31-50% ┤ 0                                        
    > 51-70% ┤ 0                                        
    > 71-90% ┤■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 1   
   > 91-100% ┤ 0                                        
             └                                        ┘ 
 12.261641 seconds (16.78 M allocations: 860.561 MiB, 1.42% gc time, 76.49% compilation time: 10% of which was recompilation)

Do see something similar in your run or is this screen missing? I was also wondering whether you're still experiencing the memory issue or if it has been solved by reducing the number of threads. Thanks!