SnakeMake best practices

[reid-a]

Snakemake has options to use "profiles", as well as the use of YAML files, to control interaction with clusters, which inform the best practices for running on an HPC cluster. This lesson should examine these best practices with a view to doing this the right way, aligned with the SnakeMake community.

[reid-a]

Related: SnakeMake also comes with a linter, and there are more general best practices in the documentation, which we should review and follow to the degree they make sense.

[ocaisa]

Regarding the use of profiles, this is a bit of a rabbit hole as to get snakemake to query the status of jobs you need a complex setup. This is solved by using cookie cutter profiles (as suggested by @vinisalazar) but that's a complex topic to teach so early in the lesson. It also impacts our portability a bit, since obviously we'll need different profiles for different schedulers.

A good example of why you want this is at snakemake/snakemake#1164

[ocaisa]

See https://github.com/Snakemake-Profiles/slurm for the Slurm profile

[ocaisa]

This looks more promising for our use case https://github.com/jdblischak/smk-simple-slurm

[bkmgit]

Profiles are helpful to avoid typing -c 1 at every invocation. Is Slurm something to emphasize throughout the lesson or enable flexible use both locally and on remote machines? One can also use Terraform to use cloud resources.