feat(herd): add a user-facing API for external resource references

[ehennestad]

Motivation

Problem — A value in an NWB file is often a term that means something specific: a species name, a brain region, a unit. HERD (HDMF External Resources Data Structure) is the schema type that records what those terms refer to in an external resource such as an ontology, so the values are standardized rather than free text. MatNWB generated the HERD class but gave no way to use it: adding one annotation meant assembling six compound tables by hand and wiring them together with zero-based row indices, and getting an annotation back out meant reversing that by hand.

Solution — Add the operations HERD is actually used for. One call records that an object refers to an external entity, and a small set of lookups reads the annotations back as MATLAB tables.

What changed

• nwb.addRef(container, ...) records that an object in the file refers to an external entity.
• nwb.getExternalResources() returns the file's HERD, creating and attaching one on first use. A file has at most one HERD, so an existing one is reused, including after nwbRead.
• HERD.toTable() flattens the six tables into one table of references, with the internal row indices resolved into the values they point at.
• HERD.getEntity, getKey, getObjectEntities and getObjectType look annotations back up.
• Displaying a HERD shows how many keys, entities, objects and files it holds, followed by the flattened table, instead of the six Data properties.

Deferred

Two pieces of the HDMF HERD API are deliberately left out of this PR:

• References to plain-value attributes (relative_path). HDMF's add_ref can target an attribute that is not a neurodata type — e.g. annotating Subject.species itself rather than the Subject — by recording a schema-relative path to it (TimeSeries.unit becomes data/unit). That path is resolved at add time, purely from the schema, through HDMF's runtime ObjectMapper, which maps every attribute of a live object back to its spec node. MatNWB has no such runtime mapping: the schema is consumed at code-generation time and nested structure is flattened into underscore-joined properties (data/unit → data_unit), which discards the property-to-spec-path relation and makes it ambiguous to reverse from the name alone. Supporting this needs a spec-walker over the cached namespace specs — a self-contained follow-up, not an architectural blocker. Until then, referencing an attribute that is itself a neurodata type (such as a column of a DynamicTable) works, other attributes raise NWB:HERD:UnsupportedAttribute rather than writing a wrong path, and MatNWB always writes relative_path empty. Files written by HDMF that already hold relative paths still read fine: toTable carries the column and getObjectType filters on it.

• The sidecar zip form (to_zip/from_zip). HDMF can save a HERD as a zip of TSV files so one set of annotations can be shared across NWB files. Not implemented here; a HERD lives only inside its file.

Implementation notes

The handwritten behaviour lives in matnwb.neurodata.HERDBase, an abstract base class attached to the generated HERD class through the customBaseClasses map in file.fillClass, following the existing DynamicTableBase and AlignedDynamicTableBase pattern.

Row indices are stored zero-based on disk to match HDMF and PyNWB, and are converted at the API boundary so no public method takes or returns one.

Semantics follow HDMF: keys are scoped to an object rather than shared across the file, so the same term on two objects is stored as two key rows; an entity keeps the URI it was first stored with and a differing URI warns; and one key may resolve to several entities. Adding a reference that is already recorded changes nothing, where HDMF appends a second identical object_keys row; both files read the same way.

HERDBase declares matlab.mixin.CustomDisplay directly. MATLAB treats a display hook defined in a class outside that lineage as an ambiguous definition rather than an override, so the mixin has to sit in it. getFooter is left to MetaClass so its missing-property warning still fires.

Examples

Recording and reading back an annotation

nwb = NwbFile('session_description', 'demo', 'identifier', 'DEMO', ...
    'session_start_time', '2018-12-02T12:57:27.371444-08:00');
nwb.general_subject = types.core.Subject( ...
    'subject_id', '001', 'species', 'Mus musculus');

nwb.addRef(nwb.general_subject, ...
    Key=nwb.general_subject.species, ...
    EntityId="NCBITaxon:10090", ...
    EntityUri="http://purl.obolibrary.org/obo/NCBITaxon_10090");

disp(nwb.general_external_resources)

Before — neither entry point exists, so the six tables have to be built by hand:

ismethod(nwb, 'addRef')                        0
ismethod(nwb, 'getExternalResources')          0
ismethod(types.hdmf_common.HERD(), 'addRef')   0

and the result displays as its raw properties:

  HERD with properties:

       entities: [1×1 types.hdmf_common.Data]
    entity_keys: [1×1 types.hdmf_common.Data]
          files: [1×1 types.hdmf_common.Data]
           keys: [1×1 types.hdmf_common.Data]
    object_keys: [1×1 types.hdmf_common.Data]
        objects: [1×1 types.hdmf_common.Data]

After

  HERD with 1 key(s), 1 entity(ies), 1 object(s), 1 file(s)
                 file_object_id                                object_id                    object_type    relative_path      field             key                entity_id                             entity_uri
    ________________________________________    ________________________________________    ___________    _____________    __________    ________________    ___________________    __________________________________________________

    {'5c462de1-6b45-4868-9bac-126e232c14bb'}    {'8462f963-37c5-4550-8ca3-d9f5fe71faab'}    {'Subject'}     {0×0 char}      {0×0 char}    {'Mus musculus'}    {'NCBITaxon:10090'}    {'http://purl.obolibrary.org/obo/NCBITaxon_10090'}

Looking up what is annotated on one object

nwb.general_external_resources.getObjectEntities(nwb, nwb.general_subject)

Before — no such method; the objects, object_keys, entity_keys and entities tables have to be joined by hand on their zero-based indices.

After

         entity_id                             entity_uri
    ___________________    __________________________________________________

    {'NCBITaxon:10090'}    {'http://purl.obolibrary.org/obo/NCBITaxon_10090'}

How to test

workingFolder = tempname; mkdir(workingFolder); cd(workingFolder);

nwb = NwbFile('session_description', 'demo', 'identifier', 'DEMO', ...
    'session_start_time', '2018-12-02T12:57:27.371444-08:00');
nwb.general_subject = types.core.Subject('subject_id', '001', 'species', 'Mus musculus');

% Annotate the subject, and a column of a table
nwb.addRef(nwb.general_subject, Key=nwb.general_subject.species, ...
    EntityId="NCBITaxon:10090", ...
    EntityUri="http://purl.obolibrary.org/obo/NCBITaxon_10090");

electrodes = types.hdmf_common.DynamicTable( ...
    'description', 'electrodes', 'colnames', {'location'}, ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64([0; 1])), ...
    'location', types.hdmf_common.VectorData('description', 'region', 'data', {'VISp'; 'VISp'}));
nwb.scratch.set('electrodes', electrodes);
nwb.addRef(electrodes, Attribute="location", Key="VISp", ...
    EntityId="MBA:385", EntityUri="https://purl.brain-bican.org/ontology/mbao/MBA_385");

disp(nwb.general_external_resources)

% Round trip
nwbExport(nwb, 'herd.nwb');
readFile = nwbRead('herd.nwb', 'ignorecache');
disp(readFile.general_external_resources.toTable())
disp(readFile.general_external_resources.getObjectEntities(readFile, readFile.general_subject))

Unit tests:

nwbtest('Name', 'tests.unit.HERDTest')

Todo

• Manually verify each public API method

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.12903% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.36%. Comparing base (46cf771) to head (0995f9e).

Files with missing lines	Patch %	Lines
+matnwb/+neurodata/HERDBase.m	98.00%	5 Missing ⚠️
+matnwb/+utility/containsObject.m	84.61%	4 Missing ⚠️
...matnwb/+common/+validation/mustBeNeurodataObject.m	76.92%	3 Missing ⚠️

Additional details and impacted files

@@                         Coverage Diff                          @@
##           fix-empty-compound-dataset-write     #877      +/-   ##
====================================================================
+ Coverage                             95.33%   95.36%   +0.03%     
====================================================================
  Files                                   234      237       +3     
  Lines                                  8352     8660     +308     
====================================================================
+ Hits                                   7962     8259     +297     
- Misses                                  390      401      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.