Common workflow schemas

[edan-bainglass]

As part of the Swiss PREMISE project, which was born out of the first MADICES event and is now a sponsor of the MADICES series, we are looking to codify specifications for metadata schemas for common workflows, presently focusing on, but not limited to computational workflows. The AiiDA common workflows implementation provides some workflows for optimizing structure geometries and computing equation of state and bond dissociation energies. Results from such workflows were used in a comparison study across 11 codes.

However, the commonality of the workflows is w.r.t the selected code, not the engine itself. In other words, the workflows are assumed to be executed by AiiDA. It is in the interest of PREMISE to remove this tight coupling, extending the common workflows to any engine. Considering that in MADICES 3 we will have workflow management system (WFMS) representation beyond AiiDA (pyiron, Galaxy, PerQueue, more?), I believe it is an opportunity to develop such a framework to facilitate the sharing of workflows across our platforms.

There are some standards and tools out there that we could consider building on, such as for example the Common Workflow Language (CWL) already active in Galaxy and the RO-Crate community. Good to hear from those who have used it.

This thread is now open for discussion on this topic for those interested.

[edan-bainglass]

Questions

1. Do you already have metadata schemas?
2. If 1, do you include semantic annotations?
3. What tools do you use manage metadata?
4. Are your workflows sharable/interoperable?

[jan-janssen]

One limitation of standards like CWL is that they are based primarily on file based communication between the different workflow steps.

To remove this restriction developers from aiida, jobflow, pyiron and simstack met as part of the "Ab initio Description of Iron and Steel (ADIS2023): Digitalization and Workflows workshop" and discussed a Python Workflow Definition. While the publication is still work in progress, we aim to complete it before the MADICES 2025 workshop. Currently we have an overview website available at https://materialdigital.github.io/ADIS2023/README.html with the corresponding repository at https://github.com/materialdigital/ADIS2023 and the recent developments at https://github.com/pyiron-dev/compare-workflow-graphs

A simple example, the following Python code:

def add_x_and_y(x, y):
    z = x + y
    return x, y, z

def add_x_and_y_and_z(x, y, z):
    w = x + y + z
    return w

x, y, z = add_x_and_y(x=1, y=2)
w = add_x_and_y_and_z(x=x, y=y, z=z)

can be represented as workflow using the Python Workflow definition:

nodes_dict = {
    0: add_x_and_y,
    1: add_x_and_y_and_z,
    2: 1,
    3: 2,
}

edges_lst = [
    {'target': 1, 'targetHandle': 'x', 'source': 0, 'sourceHandle': 'x'},
    {'target': 1, 'targetHandle': 'y', 'source': 0, 'sourceHandle': 'y'},
    {'target': 1, 'targetHandle': 'z', 'source': 0, 'sourceHandle': 'z'},
    {'target': 0, 'targetHandle': 'x', 'source': 2, 'sourceHandle': None},
    {'target': 0, 'targetHandle': 'y', 'source': 3, 'sourceHandle': None},
]

aiida, jobflow and pyiron all three provide adaptors to import and export workflows to the Python Workflow Definition. Still this is currently restricted to DAGs without dynamic modifications to the graph.

Python Workflow Definition: The python workflow definition is the current working title for this project. At the current stage it is primarily a study to validate that it is possible to define such a universal format which can be supported by multiple workflow frameworks based on the Python programming language. We are not fixed on the syntax and we do not expect anybody to write the Python Workflow Definition by hand, rather an existing workflow framework would export the workflow in the Python workflow definition and another framework would read this general workflow and execute the corresponding code.

A more materials science specific workflow example is available as well. Here the calculation of an energy volume curve with quantum espresso is defined with jobflow https://github.com/pyiron-dev/compare-workflow-graphs/blob/main/jobflow_to_universal_2.ipynb and afterwards executed with pyiron https://github.com/pyiron-dev/compare-workflow-graphs/blob/main/universal_qe_to_pyiron_base.ipynb .

It would be great to discuss these concepts in a larger group as part of MADICES 2025. In terms of the question, there is no meta data included at the moment, rather the current development primarily focused on the share ability and interoperability.

[jan-janssen]

@elinscott You are replacing the Python functions with Javascript functions, correct? I guess this works fine for these simple examples but obviously in many scientific workflows these Python functions could be more complex like a machine learning algorithm. Is it also possible to call the python functions directly in CWL? Or more specifically is it possible to load the Python functions from the module named simple_workflow.py?

[elinscott]

@jan-janssen Yes, correct. For more complicated examples, what you could do following my example above is to instead replace these ExpressionTools with Operations e.g. add_x_and_y.cwl could become

cwlVersion: v1.2
class: Operation

inputs:
  x: int
  y: int

outputs:
  z: int

i.e. this only defines the interface. Then when distributing the workflow you would additionally provide a mapping from Operations to the corresponding python functions and make it the runner's job to substitute in the concrete implementation. (Of course, standard cwl runners, such as cwltool, won't support this; the workflow could only be run by runners that are aware of and implement this mapping)

[elinscott]

So to answer your question, no, it is not possible to call python functions "directly in CWL", but this question kind of misses the point. The .cwl files dictate the logic/connectivity of the workflows, while CWL runners (such as cwltool are ultimately responsible for parsing those files and running the workflow, and these runners can call python functions directly. You would have different cwl runners for AiiDA, jobflow, etc, as each of these would want to run the workflow in their own specific way, and it is these runners that would be responsible for calling the python functions

[jan-janssen]

@elinscott I wrote a small python wrapper to call the Python functions from CWL and transfer the input and output objects in files by serializing them using pickle: https://github.com/jan-janssen/python-cwl-workflow/ . Still for me the overhead of creating a separate .cwl for each Python function highlights the advantage of a native Python-based workflow language.

[elinscott]

@jan-janssen very interesting! Good to see it can be done although yes the .cwl file for each function is clearly too clunky to be practical. (This is why I suggested using Operation objects for python functions.)

[edan-bainglass]

@jan-janssen we will 100% have a working group on this topic. We are presently working on two PREMISE project deliverables (D3.3, D4.2) that try to provided schemas and specifications for input/output of common workflows, as well as a domain-specific language for assembly/cycling of batteries (@PeterKraus Nukorn is building on tomato schemas for D3.3).

I will share more once the schemas are in better shape :)

[edan-bainglass]

We are already using RO-Crate for interoperability. Has anyone looked into Workflow Run RO-Crate flavors for describing/sharing workflows? This is developed under the WorkflowHub project, I believe. At a low level, they are using CWL. We should maybe keep this in mind. If a better-than-CWL approach is agreed upon, we could consider introducing a Crate extension/profile. The RO-Crate community is large and growing fast, already adopting many of its tools! For example, the Galaxy workflow system, of which representatives will be attending MADICES 3.

[jan-janssen]

I updated the Python Workflow Definition and included a JSON representation of the energy volume curve workflow with quantum espresso. From my perspective it could be extended to include the meta data defined by RO-Crate.

The current focus of the Python Workflow Definition is primarily the interoperability between different workflow frameworks, so in the repository above we demonstrate two workflows and how they can be transferred between two workflow frameworks, namely jobflow and pyiron_base. Rather than having a direct one to one interoperability we define a general format, so more codes can be added and directly communicate with all existing workflow frameworks which support the general format.

In summary, for me the two approaches are complementary and one goal for the working group could be combining both.

[jan-janssen]

Benchmarks for workflows in materials science

While the definition of a general workflow format is still work in progress, I think another important aspect of such a workflow working group would be the collection of example workflows. Currently I am only aware of one work which implements the same materials science workflow in different workflow frameworks - specifically to materials science. Publication, Repository

The currently tested codes in this benchmark are: aiida, cwl, gwl, kadistudio, nextflow, pydoit, pyiron, snakemake

Does anybody know about other benchmark workflows for the field of materials science? Those could be a good base for developing the general format.

[simontaurus]

prefect is not a scientific workflow engine, but the decoration of dataclass-typed function seems to be the common ground for workflow definitions.

@flow
def my_node(param: MyInputClass) -> MyOutputClass:
  ...
  return MyOutputClass(...)

The also integrates well with Object-Oriented Linked Data (OO-LD) and the semantic AST of the Abstract Workflow Language (AWL) we are working on

[edan-bainglass]

@jan-janssen are you aware of the abstract workflow language (AWL) schema?

[jan-janssen]

Yes, the use of the abstract syntax tree is definitely an interesting application.

[simontaurus]

opened an issue/discussion on this: pythonworkflow/python-workflow-definition#127

[edan-bainglass]

@scman1 great if we can have your comments on this discussion as well. We are planning a dedicated working group on the subject of generalized workflow schemas for standardized exchange and storage. On the topic of storage in ELNs, @JFRudzinski presented such efforts from NOMAD (FAIRmat), which was quite nice and also the direction in which AiiDA is going for workflow visualization (https://aiida-workgraph.readthedocs.io/en/latest/).

[JFRudzinski]

Not sure what the context is here...but I just wanted to mention in case it is useful (since it was not in my talk): Within NOMAD's ELN framework, there is a functionality for building workflows using the "general" components(classes) that are familiar to the user from the other ELN functionalities. In short, this is achieved by creating a series of "processes". Then, NOMAD performs a mapping from this series of classes to our more general NOMAD workflow schema, so that it is unified with other types of workflows stored in NOMAD.

[edan-bainglass]

@JFRudzinski do you guys have a live demo of this somewhere? I see it in the docs, but great to see it live (as shown in your DPG slides)

[JFRudzinski]

In terms of a video of the workflow visualizer:

• see slide 17 of my DPG slides
• see first video of part 3 of our DPG tutorial

Perhaps more interesting, you can play with the visualizer yourself in one of the workflow entries on the central repo:

• MD equilibration workflow
  • If you click on the label above one of the tasks, it will take you to the corresponding overview page of that task, which itself has a subworkflow visualizer at the bottom
• Custom Polarization workflow

I don't have a demo of the creation of a workflow directly from the ELN interface though, which is a bit different in terms of how the workflow entry is created (in the GUI instead of programmatically). I will see if one of my colleagues has one though.

[edan-bainglass]

The AiiDA team has been interested in common workflows for some time now, with increasing interest in workflow schemas. I'll let those involve comment (@superstar54, @GeigerJ2, @agoscinski, @mbercx)

[simontaurus]

With AWL we follow an approach to extract RDF directly from python code without the requirement for custom annotations

OO-LD flavoured pydantic class type annotations directly provide planning and validation capabilities based on the generated RDF: https://github.com/OO-LD/awl-schema?tab=readme-ov-file#planning-of-workflows

Seemless translation to Human-in-the-loop workflows be using the pydantic class type annotations to auto-generate input forms: https://github.com/OO-LD/awl-python/blob/main/examples/human_in_the_loop_async.ipynb

[schaarj]

Given the discussion during MADICES 2025, during the workshop we try to focus on following issues mostly centred around the Python Workflow definition (PWD)

Active Topics

• Adding while Loops to the PWD - moving away from the DAG
• Integrating PWD support into PerQueue
• WrapPWD in Dagster
• Implement a NOMAD Parser/Schema for the PWD
• Provide solutions for Remote Connections/Automated triggering of Workflows on remote Systems

Additional topics to be addressed if time permits

• Provide a Generic Interface/Exchange Format for data storage / Schema Mapping from data generated during a workflow run ()

Expected Deliverables at the end of the workshop

• new definition of PWD that supports dynamic workflows with while loops with examples
• PWD support in PerQueue
• PWD importer for SimStack
• NOMAD PWD Parser/Schema
• Example support for PWD in Dagster

[schaarj]

details are reported in discussion #21

[ml-evs]

Not sure if useful but we stumbled across this when looking for other ro-crate profiles: https://www.researchobject.org/workflow-run-crate/

[edan-bainglass]

Thanks @ml-evs. Abraham also mentioned that Galaxy was using the Workflow RO-Crate profile. Good to have a look at both.