Empty sigs being generated and uploaded to S3

[luizirber]

First mentioned in sourmash-bio/branchwater#24 (comment), but got new reports for empty sigs downloaded from wort:

• https://wort.sourmash.bio/view/sra/ERR12947475
• https://wort.sourmash.bio/view/sra/SRR28060683

Ran find /data/wort/wort-sra/ -size -21c -type f > empty_wort_20250204 again, 24m42s, now has 142,953 sigs 😞

---

Another one:

• https://wort.sourmash.bio/view/sra/ERR1248659 from bad signature - ERR1248659 branchwater#8

This one is not present in the list above, because it was already removed locally and S3, but not from farm, so the sig is still downloaded.

---

Plan to solve it:

• Remove empty sigs from farm and S3
• Add check for empty sigs in wort worker, don't upload if empty
• Clean up database so /view route says dataset is not computed yet (instead of redirecting to a 404 on farm), and trigger recompute

[luizirber]

OK, #77 fixes the "let's not make the problem worse" by avoiding uploading empty sigs in the future. It incidentally also fixes the daily checks for new datasets in SRA and weekly checks on NCBI assemblies, it was broken because a dep on flask-shell-ipython was in the wrong pixi environment 😓

Next: remove empty sigs from

• farm: for f in $(cat empty_wort_farm); do rm $f; done; -> 142,953 files
• S3: for f in $(cat empty_wort_farm); do s5cmd rm "s3://$f"; done; -> 142,953 objects

empty_wort_farm looks like this:

wort-sra/sigs/SRR27017016.sig
wort-sra/sigs/ERR11844198.sig

---

a fun one: SRR27017016 has a big warning, it contains human data!

This SRA run contains human sequence

Request access to [phs002199/GRU](https://dbgap.ncbi.nlm.nih.gov/aa/wga.cgi? consent=GRU&adddataset=phsphs002199) dataset through the dbGaP authorized access system.

so it should definitely fail, and not upload a sig!

---

Another one that errored out: https://trace.ncbi.nlm.nih.gov/Traces/index.html?view=run_browser&acc=ERR8085165&display=metadata

Note how this one has no SRA files, only the original Nanopore data (under Data Access)

Sure enough, if you try to run it:

❯ fastq-dump --disable-multithreading --fasta 0 --skip-technical --readids --read-filter pass --dumpbase --split-spot --clip -Z ERR8085165
2025-02-05T03:35:20 fastq-dump.3.0.10 err: name not found while resolving query within virtual file system module - failed to resolve accession 'ERR8085165' - no data ( 404 ); ncbi_phid=939BF77F75466E55000026F343DC06C0.1.m_2
2025-02-05T03:35:22 fastq-dump.3.0.10 err: name not found while resolving query within virtual file system module - failed to resolve accession 'ERR8085165' - no data ( 404 ); ncbi_phid=939BF77F75466E5500003BF3477A21BB.1.m_6
2025-02-05T03:35:22 fastq-dump.3.0.10 err: item not found while constructing within virtual database module - the path 'ERR8085165' cannot be opened as database or table
fastq-dump quit with error code 3

[luizirber]

Recompute triggered:

[luizirber]

recompute script looks like this:

import sys
import time
from pathlib import Path

import requests
from requests.auth import HTTPBasicAuth
from requests.exceptions import ConnectionError

filename = sys.argv[1]
SRA_IDS = Path(filename).read_text().splitlines()

url = "https://wort.oxli.org"
PASSWORD = "xxx"

token = requests.post(
    url + "/v1/auth/tokens", auth=HTTPBasicAuth("luizirber", PASSWORD)
).text

for sra_id in SRA_IDS:
    retry = 3
    while retry:
        try:
            r = requests.post(
                f"{url}/v1/compute/sra/{sra_id}?recompute=true",
                headers={"Authorization": f"Bearer {token}"},
            )
            if r.status_code == 401:
                token = requests.post(
                    url + "/v1/auth/tokens", auth=HTTPBasicAuth("luizirber", PASSWORD)
                ).text
                r = requests.post(
                    f"{url}/v1/compute/sra/{sra_id}?recompute=true",
                    headers={"Authorization": f"Bearer {token}"},
                )
            try:
                if "Signature already calculated" not in r.json():
                    print(sra_id)
            except:
                pass

            retry = False
        except ConnectionError as e:
            time.sleep(10)
            retry -= 1

invoked with pixi run python machine/wort-web/recompute.py machine/wort-web/empty_wort_20250204.ids
after preparing the .ids file with

sed -E 's|/data/wort/wort-sra/sigs/(.*).sig|\1|g' empty_wort_20250204 > empty_wort_20250204.ids

[luizirber]

Chugging along:

The bumps are daily SRA updates being added to the queue.

That's about 2GiB/h in signatures being added to wort.