Dynamic task resources docs => input directory size

[nick-youngblut]

New feature

The Dynamics task resources states:

Task resources can also be defined in terms of task inputs. For example:

process hello {
    memory { 8.GB + 1.GB * Math.ceil(input_file.size() / 1024 ** 3) }

    input:
    path input_file

    script:
    """
    your_command --here
    """
}

In this example, each task requests 8 GB of memory, plus the size of the input file rounded up to the next GB. This way, each task requests only as much memory as it needs based on the size of the inputs. The specific function that you use should be tuned for each process

However, the case of input directories is not covered. As far as I can tell, one cannot simply using input_dir.size() to get the total size of all files in the input directory. input_dir.listFiles().sum{ it.size() } also does not work.

If there is no way to get the total size of an input directory, then this limitation should be explicitly stated, given that the current docs state "Task resources can also be defined in terms of task inputs.", and not "task input files".

[nick-youngblut]

At least when using GCP Batch, the following results in file sizes of 0 GB:

process hello {
    memory { input_file.size() / 1024 ** 3 }

    input:
    path input_file

    script:
    """
    your_command --here
    """
}

[bentsherman]

You should be able to use the eachFileRecurse() method to recursively calculate the directory size. Check out the Path methods for details.

The snippet you referenced is only meant to be an example, not an exhaustive listing of everything you could do. You should be able to use the standard types and their methods to achieve whatever you need.

[nick-youngblut]

The snippet you referenced is only meant to be an example, not an exhaustive listing of everything you could do

I don't think an exhaustive list is needed. The main inputs are either single files are directories, so it would be helpful to show how to handle directories.

[nick-youngblut]

I've tried eachFileRecurse() using many different approaches, and I always get a java error. Can you please provide an example?

[bentsherman]

workflow {
  println directorySize( file('.') )
}

def directorySize(dir) {
  def result = 0
  dir.eachFileRecurse { path ->
    if( path.isFile() )
      result += path.size()
  }
  return result
}

[pditommaso]

I would not recommend such pattern, traversing a path over an object storage can be an expensive operation.

[nick-youngblut]

Also, that example code doesn't work for setting dynamic resources

[bentsherman]

what do you mean? you can call the function in the directive closure

[nick-youngblut]

Given the amount of compute resources that could be saved by dynamically setting resources based on the input data, I'm surprised that most/all nf-core pipeline just use task.attempt as a simple method of handling datasets of differing sizes