How to represent underlying python scripts to define processes

[samwaseda]

I don't really know what is the current status of the workflow ontology, but @joergwa and I are currently working on the workflow recipe, and were dealing with the python scripts that define workflow nodes/processes. I refer to this development for those who don't know at all what was happening by now (note: the linked page deals with the executed workflows, but here I would like to also talk about workflow recipes).

In our current development, a workflow node is defined by a python function, which is in turn defined in a python script. The node my_node is typically imported and used in a workflow in the following way:

from pyiron_workflow import Workflow

from some_package.some_path import my_node

wf = Workflow("some_workflow")
wf.node = my_node(some_input_information)
...

At this point, you can see that there are two important pieces of information: import path (some_package.some_path) and the function name (my_node). On top of this, in many cases there is a version attached to the package, i.e. in this case to some_package¹. This is so far essential information without which the workflow basically loses its meaning because we wouldn't be able to uniquely identify the underlying function.

Now on top of this it might be important to add some metadata, such as:

• Software tools used inside (Abaqus, VASP etc.)
• Creator information (name, email address etc.?)
• License
• Algorithm
• ...

While it would be nice to have this information as well, it's not crucial to do so, because once we can identify the function, we can probably recover this information via semantikon.

@joergwa and I agreed the the recipe itself should be defined as a class, because it still has to be instantiated, but we think the underlying python functions can be individuals, because they are "the same" wherever they are loaded, but so far there is no forbidding argument to make them classes if that's required.

Footnotes

1. it does not really matter whether we consider it as the version of the package or the version of the function, because they should share it anyway. ↩

[joergwa]

OK, I see many different aspects here.
First, we need to find out, what other ontologies we can reuse, to express all the aspects.
I think SWO (software ontology) might help to express software, packages, functions and versions, etc.
For additional metadata, like creator, contact point, etc. I suggest to reuse the NFDIcore or Matwerk Ontology.

I see the recipes and their parts in the scope of PMDco workflow module.
I also see the list of relevant software (Abacus, VASP, etc.) as part of the workflow module.

Lets try some some modelling:

• a recipe class R_x is subclass of plan specification
• a workflow class (process) P_x can stato:execute the recipe R_x
• R_x can have different parts:
  • these parts can be referred to as:
    • other classes (in the case, that we currently do not know about any instances of these parts)
    • instances, e.g. in the case that we refer to an existing particular e.g. a concrete python script

A simple example:

@prefix ex: <http://example.org/wf/> .
@prefix executes: <http://purl.obolibrary.org/obo/STATO_0000102> .
@prefix planned_process: <http://purl.obolibrary.org/obo/OBI_0000011> .
@prefix recipe: <https://w3id.org/pmd/co/PMD_0060008> .

ex:P_x a owl:Class ;
  owl:equivalentClass [
    a owl:Class ;
    owl:intersectionOf ( planned_process: [
      a owl:Restriction ;
      owl:onProperty executes: ;
      owl:someValuesFrom ex:R_x ;
    ] ) ;
  ] .

ex:R_x a owl:Class ;
  rdfs:comment "some recipe" ;
  rdfs:subClassOf recipe: .

Lets go further:

• some software is a plan specfication
• some software can have part some algorithm (an algorithm here is some "method" or "paradigm", which is implemented in the software)
• a license is about some software
• some software has part some package
• some package has part some function
• some function is part of a recipe (from above)
  (- also a package or a software can be part of a recipe)
• some software has version some swo:version (symbol)

In SWO they list different software, packages and programming language "categories".
I suggest to also have a "category" like "Abaqus", "VASP", but the concrete implementations of that, are instances: e.g. "Abaqus 2021HF8"

For example:

• MySoftware subClassOf Software (the most generic category, e.g. Windows, Word, Illustrator,Abaqus)
• MySoftware_3 subClassOf MySoftware (the "generation" of the software, e.g. Windows 10)
• mySoftware_3_2_1 a MySoftware_3 (the concrete build- or release version, e.g. Windows 10 build 19044)

the version pattern with SWO:

• mySoftware_3.2.1 hasVersion ex:some_version .
• ex:some_version a swo:version_number .
• ex:some_version rdfs:label "3.2.1" .

A more complete example:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix : <http://example.org/wf> .
@prefix ex: <http://example.org/wf/> .
@prefix executes: <http://purl.obolibrary.org/obo/STATO_0000102> .
@prefix has_part: <http://purl.obolibrary.org/obo/BFO_0000051> .
@prefix has_value: <https://w3id.org/pmd/co/PMD_0000006> .
@prefix is_about: <http://purl.obolibrary.org/obo/IAO_0000136> .
@prefix planned_process: <http://purl.obolibrary.org/obo/OBI_0000011> .
@prefix recipe: <https://w3id.org/pmd/co/PMD_0060008> .
@prefix software: <http://www.ebi.ac.uk/swo/SWO_0000001> .
@prefix source_code_module: <http://purl.obolibrary.org/obo/IAO_0000096> .
@prefix symbol: <http://purl.obolibrary.org/obo/IAO_0000028> .

: a owl:Ontology ;
  owl:imports <http://www.ebi.ac.uk/swo/swo/releases/2023-03-05/swo.owl> ;
  owl:imports <https://w3id.org/pmd/co/3.0.0-rc1> .

ex:AbaqusWorkflowNode a owl:Class ;
  rdfs:subClassOf [
    a owl:Class ;
    owl:intersectionOf ( recipe: [
      a owl:Restriction ;
      owl:onProperty has_part: ;
      owl:someValuesFrom ex:Abaqus_FEA ;
    ] ) ;
  ] .

ex:Abaqus_FEA a owl:Class ;
  rdfs:label "Abacus FEA" ;
  rdfs:subClassOf software: .

ex:MyWorkflowSoftware a owl:Class ;
  rdfs:label "My Workflow Software" ;
  rdfs:subClassOf software: .

ex:P_x a owl:Class ;
  owl:equivalentClass [
    a owl:Class ;
    owl:intersectionOf ( planned_process: [
      a owl:Restriction ;
      owl:onProperty executes: ;
      owl:someValuesFrom ex:R_x ;
    ] ) ;
  ] .

ex:R_x a owl:Class ;
  rdfs:comment "some recipe" ;
  rdfs:subClassOf [
    a owl:Class ;
    owl:intersectionOf ( recipe: [
      a owl:Restriction ;
      owl:onProperty has_part: ;
      owl:someValuesFrom source_code_module: ;
    ] ) ;
  ] .

ex:R_x_that_requires_a_function a owl:Class ;
  rdfs:subClassOf [
    a owl:Class ;
    owl:intersectionOf ( ex:R_x [
      a owl:Restriction ;
      owl:hasValue ex:some_function_1 ;
      owl:onProperty has_part: ;
    ] ) ;
  ] .

ex:function_name a owl:Class ;
  rdfs:subClassOf [
    a owl:Class ;
    owl:intersectionOf ( symbol: [
      a owl:Restriction ;
      owl:onProperty is_about: ;
      owl:someValuesFrom ex:source_code_function ;
    ] ) ;
  ] .

ex:package_name a owl:Class ;
  rdfs:subClassOf [
    a owl:Class ;
    owl:intersectionOf ( symbol: [
      a owl:Restriction ;
      owl:onProperty is_about: ;
      owl:someValuesFrom ex:source_code_package ;
    ] ) ;
  ] .

ex:source_code_function a owl:Class ;
  rdfs:subClassOf source_code_module: .

ex:source_code_package a owl:Class ;
  rdfs:subClassOf source_code_module: .

ex:Abaqus_FEA_2021 a owl:NamedIndividual, ex:Abaqus_FEA ;
  <http://www.ebi.ac.uk/swo/SWO_0004000> ex:version_6.21 .

ex:function_name_1 a owl:NamedIndividual, ex:function_name ;
  has_value: "calculate_speed_fast" ;
  is_about: ex:some_function_1 .

ex:my_software_1 a owl:NamedIndividual, software: ;
  has_part: ex:some_package_1 ;
  <http://www.ebi.ac.uk/swo/SWO_0004000> <http://www.ebi.ac.uk/swo/version/SWO_2000025> ;   # has version   "2.0.0"
  <http://www.ebi.ac.uk/swo/has_license> <https://creativecommons.org/licenses/by-nc-sa/4.0/> .

ex:package_name_1 a owl:NamedIndividual, ex:package_name ;
  has_value: "tempo.speed.calculations" ;
  is_about: ex:some_package_1 .

ex:some_function_1 a owl:NamedIndividual, ex:source_code_function .

ex:some_package_1 a owl:NamedIndividual, ex:source_code_package ;
  has_part: ex:some_function_1 .

ex:version_6.21 a owl:NamedIndividual ;
  has_value: 6.21 .

<https://creativecommons.org/licenses/by-nc-sa/4.0/> a owl:NamedIndividual ;
  rdfs:label "by-nc-sa/4.0" .

[samwaseda]

Thanks! I'll try to implement it and come back to this!

[joergwa]

I think we need to add to PMDco ...

imports :

• software: http://www.ebi.ac.uk/swo/SWO_0000001
• source_code_module: http://purl.obolibrary.org/obo/IAO_0000096 .
• has version : http://www.ebi.ac.uk/swo/SWO_0004000
• has license : http://www.ebi.ac.uk/swo/has_license --- > i don't like that, since it is a subprop of iao:is_about, we might just use iao:is_about

new:

• source_code_function a owl:Class ;
• source_code_package a owl:Class ;
• function_name a owl:Class ;
• package_name a owl:Class ;

[samwaseda]

For my own better understanding:

graph LR
AbaqusWorkflowNode[ex:AbaqusWorkflowNode] -.-> bn_1(( )) -- has_part -->  Abaqus_FEA[ex:Abaqus_FEA] --> software
bn_1 --> software
MyWorkflowSoftware[ex:MyWorkflowSoftware] --> software

Loading

graph LR
P_x[ex:P_x] -.-> bn_2(( )) -- executes --> R_x[ex:R_x 'some recipe']
bn_2 --> planned_process
R_x -.-> bn_3(( )) -- has_part --> source_code_module
bn_3 --> recipe
R_x_that_requires_a_function[ex:R_x_that_requires_a_function] -. has_part .-> bn_4(( )) --> R_x
bn_4 --> some_function_1[ex:some_function_1]
function_name[ex:function_name] -.-> bn_5(( )) -- is_about --> source_code_function[ex:source_code_function]
bn_5 --> symbol
package_name[ex:package_name] -.-> bn_6(( )) -- is_about --> source_code_package[ex:source_code_package]
bn_6 --> symbol
source_code_function --> source_code_module
source_code_package --> source_code_module
some_function_1[ex:some_function_1]  --> source_code_function

some_package_1[ex:some_package_1] --> source_code_package
some_package_1 -- has_part --> some_function_1

function_name_1[ex:function_name_1] --> ex:function_name
function_name_1 -- has_value --> 'calculate_speed_fast'
function_name_1 --  is_about  --> some_function_1
my_software_1[ex:my_software_1] --> software
my_software_1 -- has_part --> some_package_1[ex:some_package_1]
package_name_1[ex:package_name_1] --> package_name[ex:package_name]
package_name_1 --  has_value --> 'tempo.speed.calculations'
package_name_1 --  is_about --> some_package_1[ex:some_package_1]

Loading

[samwaseda]

@joergwa I don't quite understand what the relationship of the ex:AbaqusWorkflowNode, ex:Abaqus_FEA and ex:MyWorkflowSoftware with the rest of the graph is. Shouldn't there be a connection between ex:AbaqusWorkflowNode and P_x?

[joergwa]

There are multiple ways.

1. ex:AbaqusWorkflowNode gets executed by P_x

   • AbaqusWorkflowNode is some Software. Software is a plan specification. A process (e.g. P_x) can stato:execute plan specifications.

2. ex:AbaqusWorkflowNode is part of some recipe

   • since AbaqusWorkflowNode ia an Information content entity, it can also be part of other information content entities

You can also follow both. (You need to define what your practice will be, and that needs to be documented.)

For me it is still a little bit confusing what the differences are between:

• a recipe
• a software, and its different kinds (MyWorkflowSoftware, Abaqus_FEA, AbaqusWorkflowNode)
• a workflow node
• a workflow
  Could you provide some definitions?

[samwaseda]

Ok let's see if I manage to deliver something meaningful here:

@meta(used=URIRef("http:///www.some_ontology.org/uri_for_abaqus_with_certain_version"))
@node
def run_abaqus(mesh):
    ...
    return results

@node
def get_volume(abaqus_results):
    ...
    return volume

@workflow
def my_workflow(mesh):
    results = run_abaqus(mesh)
    volume = get_volume(results)
    return volume

Here's a short example that (as far as I can see) uses all the components. I would define the terms

• Software: A tool used within a function which executes an underlying program following a certain algorithm, with a given license and a version number. Software is to be included into the knowledge graph only if the information is provided by the user (in this case via meta(used=...)¹)
• Function: A static callable unit as it is defined via def with a clear set of inputs and outputs as well as the execution instructions
• Node: Instantiation of a python function in the context of a workflow (i.e. each node has an underlying function)
• Workflow: A set of nodes whose inputs and outputs are connected via edges. A workflow can itself be a node.
• Workflow Manager

There is no "recipe" as metadata to be stored in the knowledge graph.

Footnotes

1. For used I'm currently using prov:used ↩

[samwaseda]

Here the same workflow in the pyiron_workflow syntax:

@meta(used=URIRef("http:///www.some_ontology.org/uri_for_abaqus_with_certain_version"))
@node
def run_abaqus(mesh):
    ...
    return results

@node
def get_volume(abaqus_results):
    ...
    return volume

wf = Workflow("my_workflow")
wf.abaqus = run_abaqus(mesh)
wf.volume = get_volume(wf.abaqus.outputs.results)

[joergwa]

Summing up our meeting from today: We need clearer definitions with distinctions to other related terms, e.g.:

• Workflow Attached Software ¹: a Software (SWO:0000001) that is used within the execution instructions of a workflow function ...
• Workflow Function: a plan specification (IAO:0000104) that represents a static callable unit (method/function) as part of a (workflow) software program, with a set of inputs and outputs as well as execution instructions, and denoted by a function name (subclass of identifier IAO:0020000).
• Workflow Node: a plan specification (IAO:0000104) which represets the application of a Workflow Function ², or a workflow (set of workflow nodes). ³
• Workflow: A Workflow Node which represents a set of other Workflow Nodes whose inputs and outputs are connected according to the workflow specification.

This kind of "recursion" is a bit delicate, I am not sure now about it.

We need the following relations:

• a replacement for "prov:used"; this property is used on "prov:activities" which are mapped to bfo:processes; but here we are actually in the subtree of "bfo:generally dependent continuants", which are disjunct with bfo:processes. We just want to represent, that in a workflow function some software is utilized for some task. We cannot use "swo:uses software" because this would imply that the workflow function is a software. We cannot use has part/part of, because the function does not have "a part", which is a software. I did not find anything appropriate, so I suggest to either use 'is about' or creating a new property (e.g. "pmd:uses software").

• the relation between workflow node and workflow function; I suggest to use "is_about" or a new subproperty of is_about, e.g. "pmd:specifies_application_of"; an alternative would be "has part". This might be also used for Workflows (as WorkflowNodes)

Summing it up; here some example triples:

node_1 a WorkflowNode .  # run abaqus node
AbacusSoftwareVersion123 isAbout node_1 .  # or "pmd:uses software"
my_function_1 a WorkflowFunction; 
my_function_1 denoted by name_1 .
name_1 a Identifier .
name_1 hasValue "run_abaqus" .
node_1 is_about my_function_1 .   # or "pmd:specifies_application_of"

node_2 a WorkflowNode .  # get volume node
my_function_2 a WorkflowFunction .
my_function_2 denoted by name_2 . 
name_2 a Identifier .
name_2 hasValue "get_volume" .
node_2 is_about my_function_2 .   # or "pmd:specifies_application_of"

my_workflow a Workflow .
my_workflow hasPart node_1 .
my_workflow hasPart node_2 .
node_1 precedes node_2 .

etc. etc.

wdyt?

Footnotes

1. needs better label here (but not smth. like "Software", or "UsedSoftware") ↩

2. each node has an underlying workflow function, multiple nodes can instantiate the same function ↩

3. I avoid the term "instantiation" because it can lead to confusion (in the ontology domain it means something different than in the programming domain); ↩

[samwaseda]

Looking into IAO, I got the feeling that their definition of software or software application etc. fits in pretty well for what I wanted for software (users must derive the subclasses according to their needs, which I guess is not a difficult task since that's anyway what's happening in SWO). From my point of view, bfo:process or obi:planned_process (cf. this page) was a pretty good choice for node. Maybe for function we can simply use iao:software_script (IAO:0000595) and we connect participates_in with node?

[samwaseda]

Actually after having written my comment above, I started liking the idea even more: function is a plan specification, and node is the process generated by a function.

[joergwa]

What about this.

There is some data, which might be about some particular quality (e.g. volume):

'volume data' $${\displaystyle \sqsubseteq }$$ 'iao:data item' and ('is about' some volume)
'abaqus data' $${\displaystyle \sqsubseteq }$$ 'iao:data item'
'mesh data' $${\displaystyle \sqsubseteq }$$ 'iao:data item'

We introduce a new class for workflow definitions. And below that, we introduce the new one for our example which also has the information about what functions are to be used and in which order:

'workflow definition' $${\displaystyle \sqsubseteq }$$ 'iao:plan specification'
'my workflow' $${\displaystyle \equiv }$$ 'workflow definition'
   and ('has part' some 'get volume')
   and ('has part' some ('run abaqus' and (precedes some 'get volume')))

The workflow functions are referring to nodes which have particular (data) inputs and outputs (we do not extra subclass the 'workflow node').

'workflow function' $${\displaystyle \sqsubseteq }$$ 'iao:software method'
'run abaqus' $${\displaystyle \equiv }$$ 'workflow function'
   and ('is about' value 'Abacus Version 1234')
   and ('executed by' some
      ('workflow node'
      and ('has part' some
         ('input assignment'
         and ('has participant' some 'mesh data')))
      and ('has part' some
         ('output assignment'
         and ('has participant' some 'abaqus data')))))

'get volume' $${\displaystyle \equiv }$$ 'workflow function'
   and ('executed by' some
      ('workflow node'
      and ('has part' some
         ('input assignment'
         and ('has participant' some 'abaqus data')))
      and ('has part' some
         ('output assignment'
         and ('has participant' some 'volume data')))))

On the occurrent side we define the node, which is connected to a function:

'workflow node' $${\displaystyle \equiv }$$ 'planned process'
   and (executes some 'workflow function')

We define a workflow run, which is connected to a workflow definition:

'workflow run' $${\displaystyle \equiv }$$ 'planned process'
   and (executes some 'workflow definition')

And this is the one, for our example, which is connected to our specification:

'my workflow run' $${\displaystyle \equiv }$$ 'workflow run'
   and (executes some 'my workflow')

The full example is here (rename *.txt to *.ttl):
example3.txt

[samwaseda]

ok this sounds good to me. Let's give it a try.

[samwaseda]

Do you think there's also a possibility to have the information about inputs and outputs of the python functions as well? In particular, it's mesh, results etc. in my example above. I know that for the case for node we defined process boundaries for them, but for the python function we need to treat them differently, right?