CF Roadmap: Discovery metadata

[feggleton]

Topic for discussion

Topic for discussion

At this year's CF Workshop we divided the roadmap preparations into multiple themes. Discovery metadata was of particular interest to a lot of people and how this type of information should be captured through CF, if at all. If you're interested in this topic please comment below! The roadmap publication will include CF's future direction on this topic or direct users to other existing standards which should be used. I've written a few questions to spark discussion:

• Where should discovery metadata be recorded in CF?/how should it be recorded?
• Should discovery metadata be recorded in CF-NetCDF? or links provided to discovery metadata records ie. WCMP2 (WMO Core Metadata Profile standard used for creating metadata records in WIS2 catalogue for example)
• Does anything new need to be introduced? OR can we point towards existing standards i.e. ACDD

[lhmarsden]

I don't think ACDD is being developed - last update 2014 I think I read somewhere.

ACDD is okay for human-readability but some terms and descriptions could be developed to improve machine-readability. At MET Norway Arctic Data Centre, we have added to the descriptions, and have requirements instead of recommendations: https://adc.met.no/submit-data-as-netcdf-cf

The impression I get is that CF do not want to integrate ACDD into CF, but instead link to a certain version of an independently maintained convention. It will be interesting to see what approach is taken with provenance in this regard, as this might set the precedent on how to proceed with discovery metadata too:
https://github.com/orgs/cf-convention/discussions/389

If CF decide not to 'take on' ACDD or include their own recommendations for discovery metadata, then perhaps this does not need to be a section of the roadmap paper? Though it would be good if the collective voice of CF could encourage further development in discovery metadata conventions.

[feggleton]

I agree, there is a lot of similarity between this approach and provenance metadata. That issue has had a lot of discussion so I'll be interested to follow their conclusions. It may be that these topics are discussed as one perhaps. At the Met Office we are choosing to keep discovery metadata separate to the files, held in a stand alone JSON record, following a schema very similar to WMOs Core Metadata Profile.

[lhmarsden]

We also keep separate metadata files, but I would still advocate for rich metadata in the NetCDF files in addition. If you have downloaded the file it is useful not to have to look elsewhere for this information. More so if you are passing data around via email, for example. We have tools to populate our sidecar metadata files based on the global attributes in the NetCDF file. Therefore if a data provider has created a netCDF file with rich, compliant metadata, they don't need to provide anything else for us to publish the data.

This is particularly important if you are wanting to publish thousands of files in one go (which can be useful from a granularity perspective).

[feggleton]

I agree, we also extract metadata from NetCDF files which map to the properties in our discovery metadata schema so I agree some information is useful alongside the data. I guess the question is, does CF introduce new attributes to capture this information? My concern, after having created a discovery metadata specification for our organisation (following other standards) is that there are a lot of properties to capture. My preference would be to recommend a standard which provides suggestions for attributes which could be included. But as you say, ACDD is not being maintained (although I can see activity from 2023 https://wiki.esipfed.org/ACDD_1-3_Follow-on_Discussions). I can't find any other standards for NetCDF for discovery right now.

[davidhassell]

Hello,

The second half (24-25 September) of the 2025 CF workshop is dedicated to the CF roadmap, including hackathons dedicated to advancing the road map topics. These hackathons have no pre-determined structure and can be used for anything from further debate to writing an actual proposal - whatever would be most useful the topic at this time.

If there is interest in working on this roadmap topic in this way, please let us know in a comment on this discussion soon, so that we can advertise it on the meeting agenda - this is a good opportunity to possibly bring in fresh ideas from people who may not yet have engaged with the roadmap.

Thanks, on behalf of the workshop committee.

[efisher008]

Hi @feggleton @lhmarsden,

It would appear that you both have been the most active in this discussion. Would either or both of you have interest in leading a hackathon on this topic at the CF Workshop, or reporting an outcome of the discussion for the roadmap?

Thanks and best regards,
Ellie

[lhmarsden]

Due to the timing of the workshop (17-20 for me in Norway) I will be unable to chair anything. I am sharing my house with my 7 year old.

[feggleton]

I do not believe there is enough interest in this topic for a hackathon discussion as seen above. I think there isn't anything for CF to pursue in this area right now. I am working heavily in the discovery metadata area outside of CF and there is a lot of activity happening which I don't think CF needs to be involved in right now (and noone is asking for it) or anything CF needs to implement to facilitate this work right now. Therefore I don't think we need a discussion but I would be happy to write something in the document when it's created.

[feggleton]

@efisher008 it would appear a discussion about this topic already took place in this discussion #294 but I had missed it. The discussion came to a similar conclusion about the use of ACDD and that CF need not add anything but point towards ACDD as a recommendation for discovery metadata.