Add support for cancer samplesheets

[ericblanc20]

Is your feature request related to a problem? Please describe.
At the moment, cancer sample sheets are poorly supported in cubi-tk (see examples below)

Describe the solution you'd like
Better support for cancer projects including multi-assays (WGS, WES & mRNA_seq are frequent, possibly also including single cell data)

Describe alternatives you've considered
Replacing biomedsheets by some simple & flexible 3rd party sample description (NOT isa-tab, then)

Examples of missing support for cancer samplesheets

• snappy/parse_sample_sheet.py only deals with germline sample sheets (all parsing is done through family ids)
• Same with snappy/pull_sheets.py & with snappy/check_local.py.
• In sodar/pull_raw_data.py, the on_visit_material method of the LibraryInfoCollector class hard-codes references to family ids, which are limited to germline samplesheets.
• In snappy/itransfer_common.py, the IndexLibrariesOnlyMixin class is only implemented for germline cases
• The snappy/itransfer_sv_calling.py & snappy/itransfer_variant_calling.py are only valid for germline (Note that snappy/itransfer_step.py is an attempt to transfer the output of any step to SODAR. If properly extended & tested, it could replace the other snappy/itransfer_xxx.py files).

[ericblanc20]

Examples of problems using cubi-tk on cancer data

Configuration

The example data set is RASi Resitance (project UUID 0cac37c4-6a00-47aa-9488-a04b53348dc6, assay UUIID f8a8ec38-0968-42ec-b336-5bca47852421).

There are 2 cell lines & 6 samples in total (2 normals & 4 tumors)

The snappy configuration file is

static_data_config:
  reference:
    path: /data/cephfs-1/work/groups/cubi/projects/biotools/static_data_by_ref/GRCh38.d1.vd1/reference/GRCh38.d1.vd1.fa
  features:
    path: /data/cephfs-1/work/groups/cubi/projects/biotools/static_data_by_ref/GRCh38.d1.vd1/annotation/GENCODE/33/gencode.v33.primary_assembly.annotation.GRCh38.d1.vd1.gtf

# Step Configuration ==============================================================================
step_config:

  hla_typing:
    tools: [optitype]

# Data Sets =======================================================================================
data_sets:
  exomes:
    sodar_uuid: 0cac37c4-6a00-47aa-9488-a04b53348dc6
    file: SampleSheet.tsv
    search_patterns:
    - {"left": "*_R1_001.fastq.gz", "right": "*_R2_001.fastq.gz"}
    - {"left": "*_R1.fastq.gz", "right": "*_R2.fastq.gz"}
    search_paths:
    - /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/input_links
    type: matched_cancer
    naming_scheme: only_secondary_id

cubi-tk sodar

download-sheet

$ cubi-tk sodar download-sheet --yes 0cac37c4-6a00-47aa-9488-a04b53348dc6 dest
[I 250210 17:59:30] Starting cubi-tk sodar download-sheet
[I 250210 17:59:30]   config: Config(config=None, verbose=False, sodar_server_url=None, sodar_url='https://sodar.bihealth.org/', sodar_api_token=***, makedirs=True, overwrite=False, dry_run=False, yes=True, show_diff=False, show_diff_side_by_side=False, project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6', output_dir='dest')
[I 250210 17:59:31] Writing dest/i_Investigation.txt
[I 250210 17:59:31] About to write file contents to dest/i_Investigation.txt
[I 250210 17:59:31] Writing dest/s_rasi_perturbation.txt
[I 250210 17:59:31] About to write file contents to dest/s_rasi_perturbation.txt
[I 250210 17:59:31] Writing dest/s_rasi_resistance.txt
[I 250210 17:59:31] About to write file contents to dest/s_rasi_resistance.txt
[I 250210 17:59:31] Writing dest/a_rasi_perturbation_mass_cytometry.txt
[I 250210 17:59:31] About to write file contents to dest/a_rasi_perturbation_mass_cytometry.txt
[I 250210 17:59:31] Writing dest/a_rasi_resistance_exome_sequencing.txt
[I 250210 17:59:31] About to write file contents to dest/a_rasi_resistance_exome_sequencing.txt
[I 250210 17:59:31] Writing dest/a_rasi_resistance_mass_cytometry.txt
[I 250210 17:59:31] About to write file contents to dest/a_rasi_resistance_mass_cytometry.txt
[I 250210 17:59:31] Writing dest/a_rasi_resistance_scRNA_sequencing.txt
[I 250210 17:59:31] About to write file contents to dest/a_rasi_resistance_scRNA_sequencing.txt
[I 250210 17:59:31] All done. Have a nice day!

Works fine

pull-data-collection

$ cubi-tk sodar pull-data-collection --assay f8a8ec38-0968-42ec-b336-5bca47852421 --output-dir dest --all-files 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250210 17:56:00] Starting cubi-tk sodar pull-data-collection
[I 250210 17:56:00]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='sodar', sodar_cmd=<bound method PullDataCollection.run of <class 'cubi_tk.sodar.pull_data_collection.PullDataCollection'>>, sodar_url='https://sodar.bihealth.org/', overwrite=False, assay_uuid='f8a8ec38-0968-42ec-b336-5bca47852421', preset=None, file_pattern=None, all_files=True, substring_match=False, sample_list=None, biomedsheet=None, tsv=None, tsv_column=1, tsv_skip_rows=0, output_dir='dest', output_pattern='{collection}/{subcollections}/{filename}', output_regex=[], project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
[I 250210 17:56:00] Starting remote files search ...
[I 250210 17:56:00] Using provided Assay UUID: f8a8ec38-0968-42ec-b336-5bca47852421
[E 250210 17:56:00] Provided Assay UUID is not present in the Study.
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/__init__.py", line 125, in run
    return args.sodar_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/pull_data_collection.py", line 160, in run
    return cls(args).execute()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/pull_data_collection.py", line 220, in execute
    remote_files_dict = filesearcher.perform()
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 56, in perform
    assay_path = self.get_assay_irods_path(assay_uuid=self.assay_uuid)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 83, in get_assay_irods_path
    assay = study.assays[assay_uuid]
            ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'f8a8ec38-0968-42ec-b336-5bca47852421'

Fails with error, although the assay UUID is correct.

pull-raw-data

$ cubi-tk sodar pull-raw-data --yes --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6 dest
[I 250210 18:03:06] Starting cubi-tk sodar pull-raw-data
[I 250210 18:03:06]   config: Config(config=None, verbose=False, sodar_server_url=None, sodar_url='https://sodar.bihealth.org/', sodar_api_token=***, overwrite=False, min_batch=0, allow_missing=False, dry_run=False, irsync_threads=None, yes=True, project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6', assay='f8a8ec38-0968-42ec-b336-5bca47852421', output_dir='dest')
[I 250210 18:03:06] Using irods path of assay f8a8ec38-0968-42ec-b336-5bca47852421: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/sample_data/study_0f9fd72e-0e0a-4912-beb2-be863fa6b81b/assay_f8a8ec38-0968-42ec-b336-5bca47852421
[I 250210 18:03:07] No samples to transfer with --min-batch=0
[I 250210 18:03:07] All done. Have a nice day!

Fails, no sample is found, but no error with assay UUID.

landing-zone-create

$ cubi-tk sodar landing-zone-create --assay f8a8ec38-0968-42ec-b336-5bca47852421 --dry-run 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250210 18:05:14] Starting cubi-tk sodar landing-zone-create
[I 250210 18:05:14]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='sodar', sodar_cmd=<bound method CreateLandingZoneCommand.run of <class 'cubi_tk.sodar.lz_create.CreateLandingZoneCommand'>>, sodar_url='https://sodar.bihealth.org/', unless_exists=False, dry_run=True, assay='f8a8ec38-0968-42ec-b336-5bca47852421', format_string=None, project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
{"sodar_uuid": "cf6f7e16-5dcc-4a26-9fd5-2e9674dad433", "date_modified": "2025-02-10T18:05:14.550229+01:00", "status": "CREATING", "project": "0cac37c4-6a00-47aa-9488-a04b53348dc6", "title": "20250210_180514", "description": "", "user": {"sodar_uuid": "f8fd57cf-6364-4429-bf28-d891ab884568", "username": "blance@CHARITE", "name": "Eric Blanc", "email": "eric.blanc@bih-charite.de"}, "assay": "f8a8ec38-0968-42ec-b336-5bca47852421", "status_info": "Creating landing zone in iRODS", "configuration": null, "config_data": {}, "irods_path": "/sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514"}
[I 250210 18:05:14] All done. Have a nice day!

Works, but doesn't respect the --dry-run directive (landing zone is always created), and the collections for each library are not created (perhaps on purpose?)

landing-zone-list

$ cubi-tk sodar landing-zone-list 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250210 18:09:47] Starting cubi-tk sodar landing-zone-list
[I 250210 18:09:47]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='sodar', sodar_cmd=<bound method ListLandingZoneCommand.run of <class 'cubi_tk.sodar.lz_list.ListLandingZoneCommand'>>, sodar_url='https://sodar.bihealth.org/', unless_exists=False, dry_run=False, format_string=None, filter_status=['ACTIVE', 'FAILED', 'VALIDATING'], project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
{"sodar_uuid": "cf6f7e16-5dcc-4a26-9fd5-2e9674dad433", "date_modified": "2025-02-10T18:05:15.519439+01:00", "status": "ACTIVE", "project": "0cac37c4-6a00-47aa-9488-a04b53348dc6", "title": "20250210_180514", "description": "", "user": {"sodar_uuid": "f8fd57cf-6364-4429-bf28-d891ab884568", "username": "blance@CHARITE", "name": "Eric Blanc", "email": "eric.blanc@bih-charite.de"}, "assay": "f8a8ec38-0968-42ec-b336-5bca47852421", "status_info": "Available with write access for user", "configuration": null, "config_data": {}, "irods_path": "/sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514"}
{"sodar_uuid": "8e60e3af-816e-4dd4-958e-d7d7d7edb045", "date_modified": "2025-02-10T18:05:27.779431+01:00", "status": "ACTIVE", "project": "0cac37c4-6a00-47aa-9488-a04b53348dc6", "title": "20250210_180526", "description": "", "user": {"sodar_uuid": "f8fd57cf-6364-4429-bf28-d891ab884568", "username": "blance@CHARITE", "name": "Eric Blanc", "email": "eric.blanc@bih-charite.de"}, "assay": "f8a8ec38-0968-42ec-b336-5bca47852421", "status_info": "Available with write access for user", "configuration": null, "config_data": {}, "irods_path": "/sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180526"}
[I 250210 18:09:47] All done. Have a nice day!

Works fine

ingest-fastq

$ cubi-tk sodar ingest-fastq --remote-dir-pattern 'SW480-N1-DNA1-WES1/raw_data/2025-02-10/{filename}' /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/input_links/SW480-N1-DNA1-WES1/ cf6f7e16-5dcc-4a26-9fd5-2e9674dad433
[I 250210 18:21:19] Starting cubi-tk sodar ingest-fastq
[I 250210 18:21:19]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='sodar', sodar_cmd=<bound method SnappyItransferCommandBase.run of <class 'cubi_tk.sodar.ingest_fastq.SodarIngestFastq'>>, sodar_url='https://sodar.bihealth.org/', num_parallel_transfers=8, sync=False, yes=False, validate_and_move=False, preset='default', src_regex=None, remote_dir_pattern='SW480-N1-DNA1-WES1/raw_data/2025-02-10/{filename}', match_column=None, sample_collection_mapping=[], remote_dir_date='2025-02-10', collection_column=None, tmp='temp/', assay=None, sources=['/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/input_links/SW480-N1-DNA1-WES1/'], destination='cf6f7e16-5dcc-4a26-9fd5-2e9674dad433')
[I 250210 18:21:19] Target iRODS path: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
[I 250210 18:21:19] Computing MD5 sums for 0 files of 0.0 B with up to 8 processes
[I 250210 18:21:19] Missing MD5 files:
    
0.00B [00:00, ?B/s]
[I 250210 18:21:19] Planning to transfer the following files:
/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data/P3547_01_DNA_5_SW480_origin_S74_L008_R1_001.fastq.gz
/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data/P3547_01_DNA_5_SW480_origin_S74_L008_R1_001.fastq.gz.md5
/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data/P3547_01_DNA_5_SW480_origin_S74_L008_R2_001.fastq.gz
/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data/P3547_01_DNA_5_SW480_origin_S74_L008_R2_001.fastq.gz.md5
[I 250210 18:21:19] With a total size of 23.4 GiB
Is this OK? [y/N] y
[I 250210 18:23:49] File transfer complete.                                                                                                                                       
[I 250210 18:23:49] Transferred files will not be automatically moved in SODAR.
[I 250210 18:23:49] All done
[I 250210 18:23:49] All done. Have a nice day!

Works fine (but the collection & internal file structure are not automagically setup)

check-remote

$ cubi-tk sodar check-remote --assay-uuid f8a8ec38-0968-42ec-b336-5bca47852421 --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250210 18:28:38] Starting cubi-tk sodar check-remote
[I 250210 18:28:38]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='sodar', sodar_cmd=<bound method SodarCheckRemoteCommand.run of <class 'cubi_tk.sodar.check_remote.SodarCheckRemoteCommand'>>, sodar_url='https://sodar.bihealth.org/', base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/RAW_DATA/P3547_exome_data', filename_only=False, recheck_md5=False, report_md5sums=False, assay_uuid='f8a8ec38-0968-42ec-b336-5bca47852421', project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
[I 250210 18:28:38] Starting remote files search ...
[I 250210 18:28:38] Using provided Assay UUID: f8a8ec38-0968-42ec-b336-5bca47852421
[E 250210 18:28:38] Provided Assay UUID is not present in the Study.
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/__init__.py", line 125, in run
    return args.sodar_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/check_remote.py", line 360, in run
    return cls(args).execute()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar/check_remote.py", line 398, in execute
    remote_files_dict = irodscollector.perform()
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 56, in perform
    assay_path = self.get_assay_irods_path(assay_uuid=self.assay_uuid)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 83, in get_assay_irods_path
    assay = study.assays[assay_uuid]
            ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'f8a8ec38-0968-42ec-b336-5bca47852421'

Fails with same error as for the pull-data-collection sub-command.

cubi-tk snappy

itransfer-raw-data

$ cubi-tk snappy itransfer-raw-data --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250211 09:19:46] Starting cubi-tk snappy itransfer-raw-data
[I 250211 09:19:46]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='snappy', snappy_cmd=<bound method SnappyItransferCommandBase.run of <class 'cubi_tk.snappy.itransfer_raw_data.SnappyItransferRawDataCommand'>>, sodar_url='https://sodar.bihealth.org/', num_parallel_transfers=8, tsv_shortcut='cancer', first_batch=0, last_batch=None, base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', remote_dir_date='2025-02-11', remote_dir_pattern='{library_name}/{step}/{date}', yes=False, validate_and_move=False, assay='f8a8ec38-0968-42ec-b336-5bca47852421', destination='0cac37c4-6a00-47aa-9488-a04b53348dc6')
[I 250211 09:19:46] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:19:46] Libraries in sheet:
    DLD1-N1-DNA1-WES1
    DLD1-T1-DNA1-WES1
    DLD1-T2-DNA1-WES1
    DLD1-T3-DNA1-WES1
    SW480-N1-DNA1-WES1
    SW480-T1-DNA1-WES1
[I 250211 09:19:46] Found active Landing Zone: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
...

Works fine

itransfer-ngs-mapping

$ cubi-tk snappy itransfer-ngs-mapping --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --assay f8a8ec38-0968-42ec-b336-5bca47852421 --mapper mbcs 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250210 18:36:34] Starting cubi-tk snappy itransfer-ngs-mapping
[I 250210 18:36:34]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='snappy', snappy_cmd=<bound method SnappyItransferCommandBase.run of <class 'cubi_tk.snappy.itransfer_ngs_mapping.SnappyItransferNgsMappingCommand'>>, sodar_url='https://sodar.bihealth.org/', num_parallel_transfers=8, tsv_shortcut='cancer', first_batch=0, last_batch=None, base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', remote_dir_date='2025-02-10', remote_dir_pattern='{library_name}/{step}/{date}', yes=False, validate_and_move=False, assay='f8a8ec38-0968-42ec-b336-5bca47852421', destination='0cac37c4-6a00-47aa-9488-a04b53348dc6', mapper='mbcs')
[I 250210 18:36:34] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250210 18:36:34] Libraries in sheet:
    DLD1-N1-DNA1-WES1
    DLD1-T1-DNA1-WES1
    DLD1-T2-DNA1-WES1
    DLD1-T3-DNA1-WES1
    SW480-N1-DNA1-WES1
    SW480-T1-DNA1-WES1
[I 250210 18:36:34] Found active Landing Zone: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
Can the process use this path? [yN] y
[I 250210 18:36:52] Target iRODS path: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
[I 250210 18:36:53] Computing MD5 sums for 0 files of 0.0 B with up to 8 processes
[I 250210 18:36:53] Missing MD5 files:
    
0.00B [00:00, ?B/s]
[I 250210 18:36:53] Transferring 264 files with a total size of 184.3 GiB
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 198G/198G [1:00:10<00:00, 54.8MB/s]
[I 250210 19:37:03] Transferred files will not be automatically moved in SODAR.
[I 250210 19:37:03] All done
[I 250210 19:37:03] All done. Have a nice day!

Works fine

itransfer-step

$ cubi-tk snappy itransfer-step --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --assay f8a8ec38-0968-42ec-b336-5bca47852421 --step somatic_variant_annotation 0cac37c4-6a00-47aa-9488-a04b53348dc6 --tool mbcs mutect2 vep
[I 250211 09:29:07] Starting cubi-tk snappy itransfer-step
[I 250211 09:29:07]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='snappy', snappy_cmd=<bound method SnappyItransferCommandBase.run of <class 'cubi_tk.snappy.itransfer_step.SnappyItransferStepCommand'>>, sodar_url='https://sodar.bihealth.org/', num_parallel_transfers=8, tsv_shortcut='cancer', first_batch=0, last_batch=None, base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', remote_dir_date='2025-02-11', remote_dir_pattern='{library_name}/{step}/{date}', yes=False, validate_and_move=False, assay='f8a8ec38-0968-42ec-b336-5bca47852421', destination='0cac37c4-6a00-47aa-9488-a04b53348dc6', step='somatic_variant_annotation', tool=['mbcs', 'mutect2', 'vep'])
[I 250211 09:29:07] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:29:07] Libraries in sheet:
    DLD1-N1-DNA1-WES1
    DLD1-T1-DNA1-WES1
    DLD1-T2-DNA1-WES1
    DLD1-T3-DNA1-WES1
    SW480-N1-DNA1-WES1
    SW480-T1-DNA1-WES1
[I 250211 09:29:07] Found active Landing Zone: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
Can the process use this path? [yN] y
[I 250211 09:29:19] Target iRODS path: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/landing_zones/blance@CHARITE/rasi_resistance/rasi_resistance_exome_sequencing/20250210_180514
[I 250211 09:29:20] Computing MD5 sums for 0 files of 0.0 B with up to 8 processes
[I 250211 09:29:20] Missing MD5 files:
    
0.00B [00:00, ?B/s]
[I 250211 09:29:20] Transferring 56 files with a total size of 9.0 MiB
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.47M/9.47M [01:00<00:00, 158kB/s]
[I 250211 09:30:20] Transferred files will not be automatically moved in SODAR.
[I 250211 09:30:20] All done
[I 250211 09:30:20] All done. Have a nice day!

Works fine. Note that it may not work for all steps. There is scope for formalising the pattern of files being uploaded.

check-local

$ cubi-tk snappy check-local --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:34:34] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:34:34] Starting cubi-tk snappy check-local
[I 250211 09:34:34]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=None, cmd='snappy', snappy_cmd=<bound method SnappyCheckLocalCommand.run of <class 'cubi_tk.snappy.check_local.SnappyCheckLocalCommand'>>, tsv_shortcut='cancer', base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', project_uuids=[])
[I 250211 09:34:34] Running germline sheet checks...
[I 250211 09:34:34] Checking for parent sex consistency...
[I 250211 09:34:34] Checking for dangling parents...
[I 250211 09:34:34] Checking for family identifiers...
[W 250211 09:34:34] Found donors without family ID: DLD1, SW480
[I 250211 09:34:34] All done
[E 250211 09:34:34] Something did not work out correctly.

Fails, runs germline sheet instead of cancer.

check-remote

$ cubi-tk snappy check-remote --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250211 09:37:15] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:37:15] Starting cubi-tk snappy check-remote
[I 250211 09:37:15]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='snappy', snappy_cmd=<bound method SnappyCheckRemoteCommand.run of <class 'cubi_tk.snappy.check_remote.SnappyCheckRemoteCommand'>>, sodar_url='https://sodar.bihealth.org/', tsv_shortcut='cancer', base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', md5=False, assay_uuid='f8a8ec38-0968-42ec-b336-5bca47852421', project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
[I 250211 09:37:15] Starting remote files search ...
[I 250211 09:37:15] Using provided Assay UUID: f8a8ec38-0968-42ec-b336-5bca47852421
[E 250211 09:37:15] Provided Assay UUID is not present in the Study.
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/__init__.py", line 144, in run
    return args.snappy_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/check_remote.py", line 647, in run
    return cls(args).execute()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/check_remote.py", line 691, in execute
    ).perform()
      ^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 56, in perform
    assay_path = self.get_assay_irods_path(assay_uuid=self.assay_uuid)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 83, in get_assay_irods_path
    assay = study.assays[assay_uuid]
            ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'f8a8ec38-0968-42ec-b336-5bca47852421'

Fails with same error as for the pull-data-collection sub-command.

pull-sheets

$ cubi-tk snappy pull-sheets --tsv-shortcut cancer --base-path . 
[I 250211 09:40:32] Starting to pull sheet...
[I 250211 09:40:32]   Args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=None, cmd='snappy', snappy_cmd=<function run at 0x153bc63d1440>, base_path='.', yes=False, dry_run=False, show_diff=True, show_diff_side_by_side=False, library_types=[], first_batch=0, last_batch=None, tsv_shortcut='cancer')
[I 250211 09:40:32] Will start at .
[I 250211 09:40:32] Loading data sets from .snappy_pipeline/config.yaml
[I 250211 09:40:32] Loaded 1 data sets, 1 with SODAR UUID
[I 250211 09:40:32] Pulling for 1 datasets
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/__init__.py", line 144, in run
    return args.snappy_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_sheets.py", line 358, in run
    build_sheet(
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_sheets.py", line 298, in build_sheet
    iwalker.run(builder)
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/isa_support.py", line 275, in run
    return tuple(self.gen(visitor))
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/isa_support.py", line 269, in gen
    yield from st.gen(visitor)
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/isa_support.py", line 337, in gen
    yield from assay_traversal.gen(visitor, start_name=obj.name)
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/isa_support.py", line 381, in gen
    tmp_obj = func(obj, node_path=node_path, study=self.study, assay=self.assay)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_sheets.py", line 252, in on_visit_material
    raise Exception("Cannot infer library type from %s" % library.name)
Exception: Cannot infer library type from A4005_MM_OA_GEX

Fails, presumably because it tries to access an assay with ngs data attached (cytometry). Ideally, the command would take an --assay argument.

pull-all-data

$ cubi-tk snappy pull-all-data --base-path . --output-directory dest --dry-run --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250211 09:45:02] => will download to dest
[I 250211 09:45:02] Using cubi-tk sodar pull-raw-data to actually download data
[I 250211 09:45:02] Starting cubi-tk sodar pull-raw-data
[I 250211 09:45:02]   config: Config(config=None, verbose=False, sodar_server_url=None, sodar_url='https://sodar.bihealth.org/', sodar_api_token=***, overwrite=False, min_batch=0, allow_missing=False, dry_run=True, irsync_threads=None, yes=False, project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6', assay='f8a8ec38-0968-42ec-b336-5bca47852421', output_dir='dest')
[I 250211 09:45:02] Using irods path of assay f8a8ec38-0968-42ec-b336-5bca47852421: /sodarZone/projects/0c/0cac37c4-6a00-47aa-9488-a04b53348dc6/sample_data/study_0f9fd72e-0e0a-4912-beb2-be863fa6b81b/assay_f8a8ec38-0968-42ec-b336-5bca47852421
[I 250211 09:45:03] No samples to transfer with --min-batch=0
[I 250211 09:45:03] All done. Have a nice day!
[I 250211 09:45:03] All done. Have a nice day!

Fails, cannot find any sample, presumably because of samplesheet configuration (no --tsv-shortcut argument)

pull-processed-data

$ cubi-tk snappy pull-processed-data --tsv-shortcut cancer --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --output-directory dest --file-type vcf --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250211 09:51:03] Starting cubi-tk snappy pull-processed-data
[I 250211 09:51:03]   args: Namespace(verbose=False, config=None, sodar_server_url=None, sodar_api_token=***, cmd='snappy', snappy_cmd=<bound method PullProcessedDataCommand.run of <class 'cubi_tk.snappy.pull_processed_data.PullProcessedDataCommand'>>, sodar_url='https://sodar.bihealth.org/', tsv_shortcut='cancer', base_path='/data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38', selected_samples=None, first_batch=0, last_batch=None, output_directory='dest', sample_id=False, file_type='vcf', download_all_versions=False, overwrite=False, assay_uuid='f8a8ec38-0968-42ec-b336-5bca47852421', project_uuid='0cac37c4-6a00-47aa-9488-a04b53348dc6')
[I 250211 09:51:03] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:51:03] Starting remote files search ...
[I 250211 09:51:03] Using provided Assay UUID: f8a8ec38-0968-42ec-b336-5bca47852421
[E 250211 09:51:03] Provided Assay UUID is not present in the Study.
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/__init__.py", line 144, in run
    return args.snappy_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_processed_data.py", line 138, in run
    return cls(args).execute()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_processed_data.py", line 213, in execute
    ).perform()
      ^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 56, in perform
    assay_path = self.get_assay_irods_path(assay_uuid=self.assay_uuid)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 83, in get_assay_irods_path
    assay = study.assays[assay_uuid]
            ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'f8a8ec38-0968-42ec-b336-5bca47852421'

Fails with same error as for the pull-data-collection sub-command.

pull-raw-data

$ cubi-tk snappy pull-raw-data --tsv-shortcut cancer --dry-run --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --output-directory dest --file-type vcf --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
usage: cubi-tk [-h] [--verbose] [--version] [--config CONFIG] [--sodar-server-url SODAR_SERVER_URL] [--sodar-api-token SODAR_API_TOKEN]
               {isa-tpl,isa-tab,snappy,sodar,irods,org-raw,sea-snap} ...
cubi-tk: error: unrecognized arguments: --output-directory --file-type vcf 0cac37c4-6a00-47aa-9488-a04b53348dc6
(sak) [blance_c@hpc-cpu-77 cubi_tk]$ cubi-tk snappy pull-raw-data --tsv-shortcut cancer --dry-run --base-path /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38 --assay f8a8ec38-0968-42ec-b336-5bca47852421 0cac37c4-6a00-47aa-9488-a04b53348dc6
[I 250211 09:53:56] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:53:56] Loading configuration file and look for dataset
[I 250211 09:53:56] => will download to /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/input_links
[I 250211 09:53:56] Will start at /data/cephfs-1/work/groups/cubi/projects/2024-01-22_RASi_morkel/pipelines/GRCh38
[I 250211 09:53:57] Starting remote files search ...
[I 250211 09:53:57] Using provided Assay UUID: f8a8ec38-0968-42ec-b336-5bca47852421
[E 250211 09:53:57] Provided Assay UUID is not present in the Study.
Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/miniforge3/envs/sak/bin/cubi-tk", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/__main__.py", line 120, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/__init__.py", line 144, in run
    return args.snappy_cmd(args, parser, subparser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_raw_data.py", line 141, in run
    return cls(Config(**args)).execute()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/snappy/pull_raw_data.py", line 196, in execute
    ).perform()
      ^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 56, in perform
    assay_path = self.get_assay_irods_path(assay_uuid=self.assay_uuid)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cephfs-1/work/groups/cubi/users/blance_c/Development/saks/main/src/cubi_tk/sodar_common.py", line 83, in get_assay_irods_path
    assay = study.assays[assay_uuid]
            ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'f8a8ec38-0968-42ec-b336-5bca47852421'

Fails with same error as for the pull-data-collection sub-command.

[Nicolai-vKuegelgen]

Some comments

cubi-tk sodar pull-data-collection

Fails with error, although the assay UUID is correct.

This is a separate issue from the cancer samplesheets, it is defenitely a bug thoug. I'd recommend a separate ticket for that.

cubi-tk sodar pull-raw-data

Fails, no sample is found, but no error with assay UUID.

I actually don't know what the sodar pull-raw-data function is supposed to do, exactly. We might have to re-imagine that?

landing-zone-create

Works, but doesn't respect the --dry-run directive (landing zone is always created), and the collections for each library are not created (perhaps on purpose?)

The current old SodarAPI implementation can not automatically create collections. If the API can do that in general, I think we should change the default while switching to a newer implementation.
This is however also unrelated to cancer samplesheets.

ingest-fastq

Works fine (but the collection & internal file structure are not automagically setup)

Not sure which file structure you expect? If its the Sodar base collections, then see above.

cubi-tk snappy

pull-processed-data

Fails with same error as for the pull-data-collection sub-command.

Is there an actual use-case for this? The functions isn't named very well, but its intended purpose is to download data not processed with snappy (specifically Dragen output from the cooperation with LaborBerlin)