Simplified access to block-processing logic

[vaxenburg]

Related to our lsd-lite discussion, I think it would be great if some of the block processing logic could be similarly isolated (volara-lite?) to make it easier to set up your own custom daisy tasks?

@pattonw @sheridana

[sheridana]

hmm what specifically do you mean? It is already pretty straight forward to set up custom tasks in daisy and volara

[vaxenburg]

My thinking is that if you already know how to set up daisy tasks on cluster, you don't necessarily have to use volara as a whole -- which has some learning curve and feels a bit framework-y in my opinion. For some users, it could be easier and faster to just import the stand-alone block-processing functions (if they were available) and plug them into their own daisy-on-cluster script.

I think it's exactly the same idea as separating the lsd calculation into the light-weight lsd-lite.

[sheridana]

Ah that makes sense. So basically just a library of block functions that the user can then decide how to distribute?

[vaxenburg]

Yup yup, exactly! Either a separate light-weight library or as a part of volara, or whatever:)

[pattonw]

Yes I agree that the block processing logic should be accessible. We are most of the way there.

Current state:
BlockwiseTask is a base class that attempts to provide a minimal set of abstract methods that should be implemented for any custom blockwise task. See here for a tutorial on how to build a custom task.
One of the methods that needs to be implemented is the process_block_func that should be implemented as a contextmanager that returns a function with the signature Callable[block], we usually call this process_block.

For most tasks currently, the process_block_func looks like:

@contextmanager
def process_block_func(self):
    *args = ... # some setup
    def process_block(block):
        do_task(block, *args)
    return process_block
    ... # some teardown

For the isolated blockwise functionality you are probably most interested in just using the do_task function @vaxenburg since that is about as isolated as the functionality in volara gets at the moment.

It would be nice if we could write a generic implementation of the process_block_func so that users don't need to implement it and could just write a single static method do_task(block, *args), and thus automatically provide all blockwise code in a way that could also be accessed without using the volara framework at all. But I have been unable to figure out how to do that so far due to arbitrary arguments that may be needed with arbitrary setup and teardown code.

If you would like to submit pull requests pushing this feature forward here are the steps that would be helpful:

• The do_task(block, *args) method in every task should be standardized. We need a good name for this method since process_block is already quite overloaded. Maybe execute_block or apply_block.
  • This shouldn't be too hard since most tasks are already structured pretty close to this.
• We could go one step further and implement a library of numpy of methods that operate on in-memory arrays so that we don't need a block.
  • This level of simplification my introduce more complexity that it is worth since we rely quite heavily on read/write Rois in the block processing and simply inferring that from array shapes probably won't work well.

[sheridana]

@pattonw for the library of numpy of methods, should we do this in a separate lightweight repo so we don't make volara too bloated? that way people can just add their methods there or fork and call them in their custom volara / daisy task inside the process block function. i'm not sure how much more isolation we'd want to add before making things too complicated, but this could be a nice intermediate. as an example, some functions that could be moved from the ExtractFragments task to a separate library and just take np array and *args as input: filter_avg_fragments, get_fragments, get_seeds, compute_fragments

[pattonw]

Probably a separate repo for a set of helper functions would work best. Something like a small funlib package?

[sheridana]

Yeah that sounds good. I'm sure gunpowder could also benefit by having some node functions moved there eventually.

[vaxenburg]

Yeah, I, too, was thinking more in the direction of independent numpy methods, to the extent possible.. As opposed to setting up something that is standardized and compatible with the volara framework necessarily.