Managing output files with dynamic filenames in Galaxy tools: with and without collections

[richard-burhans]

Managing output files with dynamic filenames in Galaxy tools

Non-collection outputs

• When an output file has a static filename, we can use:

<outputs>
  <data name="output1" format="fasta" label="fasta output" from_dir_dir="output_file.fa"/> 
</outputs>

• When an output file has a dynamic filename, we can use:

<outputs>
  <data name="output1" format="fasta" label="example1 fasta output">
    <discover_datasets pattern="(?P&lt;designation&gt;output_file*)\.fa" format="fasta" assign_primary_output="true" visible="true"/> 
  </data> 
</outputs>

• When an output file has a dynamic filename, we can also use:

<outputs>
  <data name="output1" format="fasta" label="fasta output" from_dir_dir="output_file*.fa"/> 
</outputs>

As far as I can tell, there is no documentation indicating that, in Galaxy tool XML, the value of the from_work_dir attribute in tool > outputs > data and tool > outputs > collection > data tags is evaluated using glob.glob().

The current documentation here and here for from_work_dir reads:

Relative path to a file produced by the tool in its working directory. Output’s contents are set to this file’s contents. The behaviour when this file does not exist in the working directory is undefined; the resulting dataset could be empty or the tool execution could fail.

The glob.glob() is being called in lib/galaxy/metadata/set_metadata.py. Is this expected behavior? If so, can we update the relevant documentation?

Collection outputs

This discussion originated from an attempt to add a dynamically named output to an existing output collection like the following:

<outputs>
  <collection name="output_collection1" type="list" label="example output collection">
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/> 
  </collection>
</outputs>

It would seem "natural" to assume that the following would work:

<outputs>
  <collection name="output_collection1" type="list" label="example output collection">
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/>
    <data name="output3" format="fasta" label="fasta output 3">
      <discover_datasets pattern="(?P&lt;designation&gt;output_file_dynamic*)\.fa" format="fasta" assign_primary_output="true" visible="true"/> 
    </data>
  </collection>
</outputs>

Unfortuantely, this syntax is not supported. My workaround was to use the from_work_dir glob.glob() hack:

<outputs>
  <collection name="output_collection1" type="list" label="example output collection">
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/>
    <data name="output3" format="fasta" label="fasta output 3" from_dir_dir="output_file_dynamic*.fa"/>
  </collection>
</outputs>

While investigating, I also discovered that output collections only support either tool > outputs > collection > data or tool > outputs > collection > discover_datasets, but not both. The following is not flagged by planemo lint as an error, but will cause the tool to fail:

<outputs>
  <collection name="output_collection1" type="list" label="example output collection">
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/>
    <discover_datasets pattern="(?P&lt;designation&gt;output_file_dynamic*)\.fa" format="fasta" visible="true"/> 
  </collection>
</outputs>

Putting the dynamic file inside it's own collection would also work, but was not what I wanted to do:

<outputs>
  <collection name="output_collection1" type="list" label="example output collection">
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/>
  </collection>
  <collection name="single_output_collextion" type="list" label="example additional collection">
    <discover_datasets pattern="(?P&lt;designation&gt;output_file_dynamic*)\.fa" format="fasta" visible="true"/> 
  </collection>
</outputs>

An alternative that I abandoned once I found the glob.glob() hack would be to add code inside the command tag:

<command>
  ln -s output_file_dynamic*.fa well_named_fasta_file.fa
</command>
<outputs>
    <data name="output1" format="fasta" label="fasta output 1" from_dir_dir="output_file1.fa"/>
    <data name="output2" format="fasta" label="fasta output 2" from_dir_dir="output_file2.fa"/>
    <data name="output3" format="fasta" label="fasta output 2" from_dir_dir="well_named_fasta_file.fa"/>
</outputs>

[jmchilton]

xref #12052

[richard-burhans]

The following IUC tools are using the glob hack:

• checkm
• cutadapt
• deepmicro
• egsea
• funannotate
• getorganelle
• integron_finder
• kraken2
• limma_voom
• mothur
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
• pathview
• pyega3
• qiime
• quast
• read_it_and_keep
• scanpy
• semibin
  • 1
  • 2
• varvamp

[mvdbeek]

If so, can we update the relevant documentation?

At this point we don't want people to use it, so no.

but not both.

I don't think anyone tried this before / had a need to do this, and it's not clear to me why you want to do this if you can move the files into a directory and discover from there ? This seems much more pragmatic and easy to follow, all the relevant metadata is set etc.

[bernt-matthias]

the following is not flagged by planemo lint as an error, but will cause the tool to fail

Can you tell me what the error is? Guess we can add a linter?

While investigating, I also discovered that output collections only support either tool > outputs > collection > data or tool > outputs > collection > discover_datasets, but not both

Guess the best would be to use multiple discover_datasets in the collection?