feat: track origins of variable from dataset creation to inference #437

b8raoult · 2025-10-07T17:56:26Z

Description

This PR will allow tracking of the origins of variables, previously only the mars request was clearly stored:

When the dataset is created: which source was used (mars, netcdf, etc), which filters were used (rename, regrid, clip, ...)
When the dataset is used in training (combining datasets, subsetting, cutout, ...) were performed.

The outcome is available calling ds.metadata()

Example:

from anemoi.datasets import open_dataset

ds = open_dataset(
    [
        {
            "dataset": "aifs-od-an-oper-0001-mars-n320-2016-2023-6h-v6",
            "end": 2023,
            "frequency": "6h",
        },
        {
            "dataset": "aifs-od-an-oper-0001-mars-n320-2016-2023-6h-v2-precipitations",
            "end": 2023,
            "frequency": "6h",
            "rename": {"tp_0h_6h": "tp"},
            "select": ["tp_0h_6h"],
        },
    ],
    end=2023,
)

Will return for tp

{
    "type": "pipe",
    "when": "dataset-usage",
    "steps": [
        {
            "steps": [
                {
                    "config": {
                        "accumulation_period": 6,
                        "grid": "n320",
                        "levtype": "sfc",
                        "param": ["tp", "cp", "sf"],
                    },
                    "name": "accumulations",
                    "type": "source",
                    "when": "dataset-create",
                },
                {
                    "config": {"param": "{param}_{startStep}h_{endStep}h"},
                    "name": "rename",
                    "type": "filter",
                    "when": "dataset-create",
                },
            ],
            "type": "pipe",
            "when": "dataset-create",
        },
        {
            "name": "rename",
            "config": {"rename": {"tp_0h_6h": "tp"}},
            "when": "dataset-usage",
            "type": "filter",
        },
    ],
}

What problem does this change solve?

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

📚 Documentation preview 📚: https://anemoi-datasets--437.org.readthedocs.build/en/437/

…e-generator

…o feat/recipe-generator

…eat/origin

for more information, see https://pre-commit.ci

aaron-hopkinson

There are quite a few unrelated changes in this PR. It would be good to isolate this to just the changes required for origin tracking in order to make it easier to review.

Some of the new classes, particularly the Origin related ones and the Projection related ones could do with some unit tests to help make clear how they're supposed to work. At the moment, the code is a bit hard to follow without the additional context. Informative type hints might also help too.

aaron-hopkinson · 2025-10-08T12:34:10Z

pyproject.toml

  "anemoi-utils[provenance]>=0.4.32",
  "cfunits",
  "glom",
-  "jsonschema",


Why is this being removed?

aaron-hopkinson · 2025-10-08T12:36:02Z

src/anemoi/datasets/create/input/context/field.py


        return GroupOfDates(sorted(set(group_of_dates) & set(filtering_dates)), group_of_dates.provider)
+
+    def origin(self, data: Any, action: Any, action_arguments: Any) -> Any:


If we're going to have type hints, can they be more specific? e.g. action must be an object supporting the origin method.

aaron-hopkinson · 2025-10-08T12:37:15Z

src/anemoi/datasets/create/input/context/field.py

+            previous = fs.metadata("anemoi_origin", default=None)
+            fall_through = fs.metadata("anemoi_fall_through", default=False)
+            if fall_through:
+                # The field has pass unchanges in a filter


Suggested change

# The field has pass unchanges in a filter

# The field has passed unchanged through a filter

aaron-hopkinson · 2025-10-08T12:44:22Z

src/anemoi/datasets/create/input/result/field.py

        return _data_request(self.datasource)

+    @property
+    def origins(self) -> dict[str, Any]:


Suggested change

def origins(self) -> dict[str, Any]:

def origins(self) -> dict[str, list]:

(Unless self._origins eventually becomes a different object)

aaron-hopkinson · 2025-10-08T12:47:14Z

src/anemoi/datasets/create/input/result/field.py

+    @property
+    def origins(self) -> dict[str, Any]:
+        """Returns a dictionary with the parameters needed to retrieve the data."""
+        return {"version": 1, "origins": self._origins}


Do we anticipate that this will need to be a versioned object of some kind? Do we really need it, or will it just make things more complicated?

aaron-hopkinson · 2025-10-08T14:18:27Z

src/anemoi/datasets/data/components.py

+    def from_slices(cls, slices):
+        return Projection(slices)
+
+    @classmethod


Both this and the method above don't use cls. Would they be better as staticmethods, or do you envisage these depending on the class in the future?

aaron-hopkinson · 2025-10-08T14:22:22Z

src/anemoi/datasets/dates/__init__.py

-    -------
-    Iterator[datetime.datetime]
-        An iterator of datetime objects.
+    Parameters


Changes seem unrelated to the origins work – move to a separate PR?

aaron-hopkinson · 2025-10-08T14:22:50Z

src/anemoi/datasets/dates/groups.py

-
-    Returns:
-        Union[str, List[str]]: The shortened list of dates.
+    Parameters


As above - unrelated to this work

aaron-hopkinson · 2025-10-08T14:23:14Z

src/anemoi/datasets/grids.py

    # TODO: Use the one from anemoi.utils.grids instead
    # from anemoi.utils.grids import ...
-    from scipy.spatial import cKDTree
+    from scipy.spatial import KDTree


Unrelated to origins tracking

aaron-hopkinson · 2025-10-08T14:24:46Z

tests/test_classes.py

+@zarr_tests
+@not_ready
+def test_class_missing_dates_fill():
+    pass


Lots of empty tests here - maybe better to create an issue for them rather than having them in here where they'll be forgotten about

b8raoult added 30 commits May 8, 2025 16:40

feat: recipe generator

9cb8c52

update

01774dd

update

33793a6

update

920d523

update

ed5d190

fix: better handling of xarray metadata

2562baf

update

f048a4b

update

8ad5eb0

Merge branch 'fix/better-handling-of-xarray-metadata' into feat/recip…

5db7a97

…e-generator

fix: support other keys that param in rename filter

b33f3ac

Merge branch 'fix/support-other-keys-than-param-in-rename-filter' int…

71d8180

…o feat/recipe-generator

typo

6d23027

Merge branch 'fix/support-other-keys-than-param-in-rename-filter' int…

6933574

…o feat/recipe-generator

add command line

9179dae

update

79a391b

update

203e09b

update

b4433bd

update

6f3fdb0

upadte

45365a1

update

d7cc82c

update

f381b00

update

da93ad6

refactor missing

3082edf

add references

ef4a5c9

refactor

93410d5

refactor

1df0ef7

refactor

18df4eb

update

3341d4c

work on migrate

58dc8a2

work on migrate

3e180f9

b8raoult added 8 commits September 13, 2025 17:46

add doc

a917c49

add origins to metadata

39bedac

Merge branch 'feat/origin' of github.com:ecmwf/anemoi-datasets into f…

56ac3ca

…eat/origin

update

9309cfe

docs

437e4aa

add filter.rst

79f1706

add docs

ef431f8

compress origins

48c9d07

b8raoult requested a review from a team as a code owner October 7, 2025 17:56

github-project-automation bot added this to Anemoi-dev Oct 7, 2025

github-project-automation bot moved this to To be triaged in Anemoi-dev Oct 7, 2025

github-actions bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file tests labels Oct 7, 2025

b8raoult and others added 2 commits October 7, 2025 18:21

Merge remote-tracking branch 'origin/main' into feat/origin

caf5b90

[pre-commit.ci] auto fixes from pre-commit.com hooks

9a7a8b7

for more information, see https://pre-commit.ci

HCookie marked this pull request as draft October 7, 2025 18:26

b8raoult added 5 commits October 7, 2025 18:31

tidy

57901c5

tidy

9343e66

tidy

095d57d

tidy

5148b61

add origins to metadata

0cd3bf6

b8raoult marked this pull request as ready for review October 7, 2025 20:18

b8raoult changed the title ~~Feat/origin~~ feat: track origins of variable from dataset creation to inference Oct 7, 2025

remove exception

ae1bb2f

mchantry added the ATS approval needed label Oct 8, 2025

github-actions bot added bug Something isn't working enhancement New feature or request labels Oct 8, 2025

aaron-hopkinson requested changes Oct 8, 2025

View reviewed changes

github-project-automation bot moved this from To be triaged to Under Review in Anemoi-dev Oct 8, 2025


		return GroupOfDates(sorted(set(group_of_dates) & set(filtering_dates)), group_of_dates.provider)

		def origin(self, data: Any, action: Any, action_arguments: Any) -> Any:

	# The field has pass unchanges in a filter
	# The field has passed unchanged through a filter

	def origins(self) -> dict[str, Any]:
	def origins(self) -> dict[str, list]:

feat: track origins of variable from dataset creation to inference #437

Are you sure you want to change the base?

feat: track origins of variable from dataset creation to inference #437

Uh oh!

Conversation

b8raoult commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What problem does this change solve?

What issue or task does this change relate to?

Additional notes

Uh oh!

aaron-hopkinson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

b8raoult commented Oct 7, 2025 •

edited

Loading