Optional metric unit for VectorData

[h-mayorquin]

There is no standard, machine-readable way to record the physical unit of a numeric DynamicTable column. A per-row quantity (a stimulus frequency in Hz, a running speed in cm/s) has nowhere structured to declare its unit today, so it gets folded into the column name or the free-text description. The motivating case is NWBEP001's EventsTable, where an event can carry a continuous numeric payload whose value and duration have clean homes but whose unit does not. There is precedent for unit-bearing columns already: TimestampVectorData and DurationVectorData both define a unit attribute fixed to "seconds".

Two options are on the table. Adding an optional unit (text) attribute directly on VectorData or create a subtype like NumericVectorData. I think the first is the best choice: zero new type, zero migration, and a reader just checks whether the attribute is present. I think this is best as it requires less churn because the unit is optional. The only downside is the semantics as unit does not make sense in some VectorData (reference columns for example).

[oruebel]

Adding an optional unit (text) attribute directly on VectorData

I agree, since it is an optional field, and since this is a need that applies broadly across tables, I think it make sense to address this at the level of VectorData via an optional unit attribute.

Note, this would need to be addressed in the hdmf-common-schema rather than in the NWB schema.

[rly]

Related:

• Add Measurements hdmf-dev/hdmf-common-schema#63
• Integrating with MeasurementData #493
• add Measurement neurodata_type #208

I think adding a unit attribute as an optional field to VectorData is a good idea, and I think the simplicity of that is worth the downside that unit doesn't make sense for some VectorData.

A slight tangent: conversion and offset:

hdmf-dev/hdmf-common-schema#63 also suggested adding conversion and offset. That never made it to the finish line, and I don't think we should do it here, but I think it's worth deciding now whether that proposal changes the choice above:

• Option A (my preference): add unit to VectorData, defer conversion and offset. If we later want conversion/offset, add a subtype whose distinguishing feature is scaling, e.g. ScaledVectorData, that inherits unit and adds the two scaling attributes. This is consistent with today's decision and keeps the scaling mechanics off the columns that will never use them.
• Option B: one subtype with all three. Don't touch the base type; create a single subtype QuantityVectorData carrying unit, conversion, and offset.

I prefer A. The only use case I can think of for conversion and offset is storing spike waveforms in the Units table as the original int16 with conversion and offset to recover volts, which saves storage space. Existing solutions seem tolerable and demand for this change seems low.

An aside on naming the type for Option B

"MeasurementData" (from https://github.com/hdmf-dev/hdmf-common-schema/pull/63) is descriptive but not super precise for cases like: - A filter cutoff frequency (e.g., 300 Hz) or an electrode contact width (e.g., 12 um), which are settings/specifications, not strictly measurements. ("specified by the manufacturer" is different from "measured from histology or an instrument.") - Thresholds - Derived values, like a predicted firing rate

If we go with Option B, I think "QuantityVectorData" (quantity is defined in metrology, the study of measurement, as a numerical value and a unit) is technically better and covers measured, configured, and derived values.

[oruebel]

I prefer A.

I agree with Option A.

[h-mayorquin]

Following my argument above: why not add conversion, offset as unit as optional to VectorData itself? that seems easier to propagate and less churn.

Is the type separation really worth the effort? what does it bring to the table?

[rly]

My main concern is that conversion and offset apply to a limited number of use cases of VectorData and adds clutter / reduces specificity to the VectorData and all of its subtypes (including VectorIndex and DynamicTableRegion).

Limited use cases: The only use case I can think of is Units.waveforms where the waveforms are best stored in the original int16 raw values which need to be converted to volts (or milli/microvolts) for interpretation. Since there are so few use cases, I recommend creating a separate ScaledVectorData type. What other use cases do you see?

Reduced specificity: With the attributes on VectorData, a user would always need to look at conversion and offset to interpret the column values, and nothing signals which columns have them. Placing them on ScaledVectorData moves the signal into the type explicitly - readers would understand that you should read those attributes to interpret the values. It also keeps conversion and offset off of VectorIndex, DynamicTableRegion, and text columns where they don't mean anything.

I do recognize that more classes = more churn through APIs and tools though. Where this falls on that balance is subjective, but I still lean toward creating a separate ScaledVectorData type for those attributes.