Skip to content

Conversation

b8raoult
Copy link
Collaborator

@b8raoult b8raoult commented Jul 9, 2025

Description

This is a rewrite of the code that parses the recipe and invoke the sources and filters in the right order. These changes are merely a simplification of the old version and are 100% backward compatible.
The input family of actions has been refactored such that all the standalone action files, e.g. concat are consolidated into a single action.py.

Additional related additions.

  • Add facility to have a source as a parameter of an other source if needed.
  • Changes are due to code moving to their own file (i.e repeated_dates).
  • A new CLI tool anemoi-datasets recipe was added to check the change. It can be use to format the YAML and validate it (against a simple JSON schema), and migrate older YAML files to new syntax. The later is experimental, and as been tested with very old recipe files. It can be a good starting point for future migrations, if ever needed.

What problem does this change solve?

This change is in preparation of adding support for observation in anemoi-dataset[create].

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@github-actions github-actions bot added bug Something isn't working enhancement New feature or request labels Oct 1, 2025
Copy link
Contributor

@aaron-hopkinson aaron-hopkinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will approve once the comment on the copy command is resolved, because I think that will break something.

We need to either revert the change to the copy command or reopen PR #151 on anemoi-utils

Co-authored-by: Marek Jacob <[email protected]>
MeraX
MeraX previously approved these changes Oct 6, 2025
Copy link
Contributor

@MeraX MeraX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work and the consideration of my comments.

aaron-hopkinson
aaron-hopkinson previously approved these changes Oct 6, 2025
@b8raoult b8raoult dismissed stale reviews from aaron-hopkinson and MeraX via 5de2b08 October 6, 2025 10:28
@aaron-hopkinson aaron-hopkinson merged commit c5408da into main Oct 6, 2025
76 of 77 checks passed
@github-project-automation github-project-automation bot moved this from Under Review to Done in Anemoi-dev Oct 6, 2025
@aaron-hopkinson aaron-hopkinson deleted the feat/refactor-create branch October 6, 2025 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ATS approved bug Something isn't working dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation enhancement New feature or request tests

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants