Centralize workflow output configuration using `nextflow.config`

[VasLem]

In my use-case, I have the following output block:

output {
    to_publish {
        mode 'copy'
        path { tp->
         def odir = tp[0]
         def files = tp[1..-1]
         def output_root = params.OUTPUT_ROOT.replaceFirst('/\\.\$', "/")
         odir = "${odir}".replace("${output_root}", "")
         println "Publishing ${files } to ${output_root}${odir}"
         odir }
    }
}

that is copied in more than 20 places, as it constitutes the generic way of handling publishable output, through the channel to_publish .
(sidenote, I have been using params.OUTPUT_ROOT, and then outputDir=params.OUTPUT_ROOT because using workflow.outputDir does not actually look into the override in the configuration, it still returns the default value, probably there is an issue with the parsing sequence)

There are discussions during the design of the new pattern here that include a bullet "Move publish options to config". However, I was not able to find any piece of documentation that supports this was implemented after all. I tried to include it in the config with the following way:

workflow {
    output {
      to_publish {
          mode = 'copy'
          path = { tp->
            def odir = tp[0]
            def files = tp[1..-1]
            def output_root = params.OUTPUT_ROOT.replaceFirst('/\\.\$', "/")
            odir = "${odir}".replace("${output_root}", "")
            println "Publishing ${files } to ${output_root}${odir}"
            odir 
          }
      }
  }
  }

but I am getting unrecognized config option 'workflow.output.to_publish.mode' from the Nextflow Language Server.

It might be worthwhile to include a footnote about its support, or lack thereof, in the documentation.

[bentsherman]

The workflow output config options are documented here: https://nextflow.io/docs/latest/reference/config.html#workflow

[bentsherman]

You should not need to reference the base output dir at all in the output path directive. Just specify a relative path and it will publish the files relative to the base output directory

[VasLem]

Thank you for the quick reply. I am guessing then that what I am asking for is only partially supported, by setting the worfklow.output.mode, not the workflow.output.path .

The reason why I am using this way is that there are dynamically constructed output paths created, for example output paths that point to the input directory.

For example if the user supplies --input_bam input_bam, and I am using a process for indexing the bam file, then I will have to publish the bai file directly where input_bam is located, however the outputDir could be different. I cannot write directly on the same path on GCP, because of a long standing problem with using path variables containing GCP-mounted directories (they are not-writable, broken symlinks, see here)

[bentsherman]

The workflow outputs are designed to publish everything into a single directory. There may be workarounds to publish into multiple directories but they won't be pretty

You could just re-publish the input BAMs into the output directory. But I think the bigger idea is to use the index file to record the BAM/BAI for each sample, regardless of where the files happen to be stored

You can then pass the index file to downstream pipelines, and Nextflow will localize the BAM/BAI file into the same task directory when it's time to process them

[VasLem]

The single-directory output is very limiting. With that chunk of code I shared with you earlier, I am supplying the output directory as the first item in the tuple to publish. It is straightforward, and pretty I would say, I was just wondering I could move this logic somewhere more centrally. Using index files is dirty, when there are already partial results and I am processing an already existing output folder.

[bentsherman]

Using index files is dirty, when there are already partial results and I am processing an already existing output folder.

Not sure what you mean by this. The index file is designed to free you from having to worry about output directories. The index file knows where all of the outputs are, so you can just pass it to any downstream pipeline and gather files into whatever structure that pipeline needs

In other words, the params / output block encourage you to think of your pipelines as index file in -> index file out, instead of directories in -> directories out.