Multi-document validation

[clbarnes]

In rust, ome_zarr_metadata takes a similar strategy of representing and validating metadata without performing IO. However, a lot of OME-Zarr validation needs metadata from Node B in order to properly validate Node A (e.g. transformations which are paths to arrays).

The strategy I'm working on in a local branch is to allow all OME-Zarr objects to report any references they have, so that the caller can fetch those nodes' metadata however they deem appropriate, and then pass them back in for a second round of validation which only addresses the remote references. There's scope for a shallow version (where only direct references from the node of interest are gathered), or deep (where you keep expanding the graph whenever a referred-to node has its own references).

The API looks something like

@dataclass
class ContextValue:
    zarr_metadata: ZarrV3Metadata
    ome: OMEZarrMetadata

@dataclass
class ValidationContext:
    node_path: Path
    """Absolute path to this Zarr node within the store."""

    references: dict[Path, ContextValue]
    """Paths must be absolute within the store and must contain `node_path`."""

    def index_relative(self, path: Path) -> ContextValue:
        abs_path = self.node_path / path
        return self.references[abs_path]

class ReferrerMixin:
    """API for an OME-Zarr metadata object which may refer to other Zarr nodes."""
    @abstractmethod
    def gather_references(self, paths: set[Path]):
        pass

    @abstractmethod
    def validate_references(self, context: ValidationContext):
        pass

class MultiscaleImageDataset(ReferrerMixin):
    def gather_references(self, paths: set[Path]):
        paths.add(self.path)

    def validate_references(self, context: ValidationContext):
        ctx = context.index_relative(self.path)
        assert ctx.zarr_metadata.is_array()

class MultiscaleImage(ReferrerMixin):
    def gather_references(self, paths: set[Path]):
        for ds in self.datasets:
            ds.gather_references(paths)
        ...
   
    def validate_references(self, context: ValidationContext):
        dtypes = set()
        for ds in self.datasets:
            ds.validate_references(context)
            ds_ctx = context.index_relative(ds.path)
            dtypes.add(ds_ctx.zarr_metadata.data_type)
            assert len(dtypes) <= 1
            ...
        ...

# and so on for other OME-Zarr metadata classes

store_url = "https://my.zarr.store"
node_root = Path("/node/of/interest")
node_of_interest = get_zarr_node(store_url, node_root)
root_ome = parse_ome(node_of_interest.attributes)
refs = set()
root_ome.gather_references(refs)
context = ValidationContext(node_root, {node_root: ContextValue(node_of_interest, root_ome)})
for ref in refs:
    path = node_root / ref
    node = get_zarr_node(store_url, path)
    context[path] = ContextValue(node, parse_ome(node.attributes))
    # extend the graph if you want

root_ome.validate_references(context)

[tlambert03]

hey @clbarnes 👋

not sure whether this is a suggestion/feature-request/or heads up of pending PR... but thanks either way! Any improvements in either functionality, performance, or code structure are all welcome.

As you probably found, we currently use a visitor pattern for structural multi-doc validation. My reasoning was that, given that ome-zarr really only has a handful of valid node types (and doesn't look to be changing very rapidly) a "baked in" knowledge of gathering references and validating them worked well enough (i.e. no ReferrerMixin ABC):

class StorageValidator:
    def visit_image(): ...
    def visit_plate(): ...
    def visit_well(): ...
    def visit_bioformats2raw(): ...

Just for my own clarity, are you proposing an arbitrarily extensible alternative to that pattern?

[clbarnes]

A bit of all of the above!

I think that the IO-less abstraction is a useful one and it looks like the multi-document validation here does require IO. Even if yaozarrs were to provide a wrapper which did the IO for you (when the appropriate dependencies are installed), it would be nice to expose the IO-less building blocks.

The strategy I'm taking in ome_zarr_metadata has half an eye on RFC-8 Collections, which is likely to introduce, or at least clear the way for, different types of relationships between OME-Zarr metadata and Zarr nodes, as well as relationships between Zarr nodes in different stores. It also means that if/when new classes are added, the bulk of the reference-reporting and validation logic can be defined alongside the new classes, but that's more subjective as I can also see the attraction of keeping the validation logic together and distinct from the data objects.

[tlambert03]

I think that the IO-less abstraction is a useful one and it looks like the multi-document validation here does require IO

Just to clarify, so we're on the same page. Yaozarrs does not require array IO to validate multidocument metadata. It only requires fsspec for the file system abstraction. But you can fully validate a complete ome hierarchy without having zarr-python or tensorstore or anything installed. Thats clear right?

So, are you looking/proposing something that wouldn't even need to read even metadata? Like a purely in memory, api driven mechanism? (I can see the value there, just want to make sure we're on the same page about what yaozarrs currently does and requires)

In any case, please do keep playing around with this, and if you want to propose a concrete api that yaozarrs could have that would be useful for you, I'm all ears. 👍

[tlambert03]

I can also see the attraction of keeping the validation logic together and distinct from the data objects.

And yes, this is definitely a design principle here. I want the core metadata pydantic models to be trivially easy to use and reason about (it was the main thing lacking elsewhere that caused me to make yet another...). But, I also do want to provide conveniences around working with these objects (such as the tree validation and writing functions already offered). And so would love to see this offered not as an expansion to the models directly, but as functional APIs that act on the models)