Ontology Development Kit v1.6

This is a major release of the Ontology Development Kit.

Update instructions

1. Read the release notes below to know what to expect.
2. Pull the latest version of the Docker image:

docker pull obolibrary/odkfull

3. For each of your ODK-managed repository, go to the src/ontology directory and run:

sh run.sh update_repo
sh run.sh update_repo

The command needs to be run twice in a row to ensure the run.sh script itself is properly updated. Alternatively, you could also pre-emptively copy the script under a new name, and run the renamed script, e.g.:

cp run.sh run-tmp.sh
sh run-tmp.sh update_repo

⚠️ Whichever method you use, be careful that the update command is not the same as the one that was used in previous ODK versions (it’s update_repo, not make update_repo)!

Release notes

Highlights (Read this!)

• Updating an existing repository (to migrate it to a newer ODK version or to apply changes made in the ODK configuration) is now made with sh run.sh update_repo (instead of sh run.sh make update_repo). When migrating to ODK 1.6 for the first time, that command will need to be run twice.
• The SPARQL-based checks now cover the entire ontology, including the components (they previously only covered the contents of the -edit file). This may cause your workflows to break due to invalid contents in the components that previously went undetected.
• The test suite now includes a check that the ID policy file (src/ontology/$(ONT)-idranges.owl) is valid. This may cause your workflows to break if the policy file is syntactically incorrect or contains invalid ranges.

Other less drastic changes that users should still probably pay close attention to are highlighted, in the following sections, with a ⚠️ symbol.

New and updated tooling

• Base system updated to Ubuntu 24.04.
• Updated tools:
  • ROBOT version 1.9.8
  • KGCL plugin version 0.5.1
  • SSSOM plugin and SSSOM-CLI tool version 1.5.1
  • Soufflé version 2.5
• Notable updated Python packages:
  • curies 0.10.19
  • kgcl-schema 0.6.9
  • kgx 2.5.1
  • linkml 1.9.2
  • llm 0.25
  • oaklib 0.6.22
  • pronto 2.7.0
  • sssom 0.4.16
• New tools:
  • ⚠️ J2cli, a command-line tool to process Jinja2 templates, has been replaced by Jinjanator. If your custom workflows invoke the j2 tool, you will need to update them to use jinjanate instead.
  • Scala-CLI version 1.8.0. This is intended to replace Ammonite as the Scala interpreter; Ammonite is still included in ODK 1.6, but only in the odkfull image – it will be removed completely in a future ODK release.
  • Dicer-CLI, a new tool to manage the ID policy file (src/ontology/$(ont)-idranges.owl).
  • New ODK plugin, providing some extra commands for ROBOT, notably:
    • odk:subset to create subsets (replacement for owltools --reasoner-query --make-ontology-from-results and owltools --extract-ontology-subset);
    • odk:check-align to check alignment against an upper-level ontology.
  • New yq-mf tool, a YAML, JSON, and XML processor.
• Other changes:
  • ⚠️ OWLTools is now only provided in the odkfull image.
  • The sparqlprog tools are still provided, but considered deprecated and will be removed in a future release.

Makefile workflows

• The $(ont)-idranges.owl file is now checked for validity as part of the normal test suite #211.
• The test suite can now check whether the ontology is “aligned” with another, upper-level ontology (PR #1174).
  • In this context, an ontology is said to be “aligned” if all its terms are classified under terms from the upper-level ontology.
  • To enable this feature, specify the IRI of the upper ontology to align against with the upper_ontology option in the configuration file.
  • Note that if enabled, this feature will cause the test suite to fail if the ontology is not aligned.
• The stripping of annotation properties from import modules (forcefully introduced in ODK 1.5) can now be disabled if needed, by setting the import_group.strip_annotation_properties ODK option to false (PR #1206).
• ⚠️ OBO artefacts are now “clean” by default, meaning that (1) they cannot contain terms with more than one label, one definition, or one comment, and (2) non-OBO axioms (which would normally be stored in a owl-axioms header tag) are stripped.
  • This relies on the new --clean-obo option provided by ROBOT 1.9.8, see its documentation for details.
  • This is controlled by the obo_format_options ODK option, which by default is set to --clean-obo "strict drop-untranslatable-axioms".
• When invoking robot relax, SubClassOf axioms are now relaxed by default, in addition to EquivalentClasses axioms.
  • This is controlled by the new robot_relax_options ODK option, which is set to --include-subclass-of true by default.
• ⚠️ When invoking robot reduce, SubPropertyOf axioms and property chains are now taken into account to evaluate whether an axiom is logically redundant.
  • This is controlled by the new robot_reduce_options ODK option, which is set to --include-subproperties true by default.
• If a context file has been set in the ODK configuration (option use_context: true), the context file will be passed to all invocations of robot (PR #1211).
  • Related: the location of the context file is now src/ontology/config/context.json, instead of src/config/context.json (PR #1150).
• The standard Makefile now includes a new REASONER variable (set to the reasoner configured with the reasoner option in the ODK configuration file) (PR #1140).
  • This allows custom workflows in $(ont).Makefile to be sure they are always using the same reasoner as the standard Makefile.
• The release workflow can now produce release artefacts in the SQLite/SemSQL format (used notably by the Ontology Access Kit) (PR #1148).
  • To enable SQLite/SemSQL output, add the value db to the export_format option in the ODK configuration.
• New targets allow to automatically generate some documentations from DOSDP patterns (PR #1121).
• It is now possible to publish only some of the SSSOM mapping sets, instead of all of them (PR #1120).
• A SSSOM mapping set can be automatically constructed by merging other sets (PR #1208).
• Overall, several sections of the standard Makefile have been refactored to make them cleaner and more readable (PRs #1194, #1168, #1214, #1216).

Runner and Infrastructure

• The run.sh script now allows to easily share a local OAK cache with the container, by setting a ODK_SHARE_OAK_CACHE environment variable (PR #1109):
  • set ODK_SHARE_OAK_CACHE=user to automatically share the local user’s own cache (~/.data/oaklib, assuming a default Pystow configuration);
  • set ODK_SHARE_OAC_CACHE=<path/to/cache> to share a cache located at an arbitrary location.
• By default, the run.sh script will automatically configure ROBOT (and other Java programs) to use up to 90% of the total amount of memory allocated to Docker (PR #1111).
  • This allows configuring max memory settings in one place only (in the configuration of the Docker daemon).
  • This can always be overriden once and for all by setting the robot_java_args option in the ODK configuration file, or occasionally by setting the ODK_JAVA_OPTS environment variable when calling the run.sh script.
• The run.sh script now detects environment variables indicating the use of a network proxy (http_proxy, https_proxy) and ensures that Java programs within the container are aware of them (PR #1119).

Seeding/updating a repository

• When seeding a new repository, by default release artefacts are not committed to the repository (PRs #1183, #1246).
  • Pass the --commit-artefacts option (or -r) to the seeding command to force the release artefacts to be committed.
  • If the --commit-artefacts option has not been ...

ODK 1.5.4 (December 2024)

This is a point release of the Ontology Development Kit.

Changes since version 1.5.4

Software updates

• ROBOT updated to version 1.9.7
• SSSOM-Java (ROBOT SSSOM plugin and command-line tool sssom-cli) updated to version 1.0.0

Bugfixes

• Fixed bogus installation of SPARQLProg tools (#1134)
• Fixed a bug that led to the repository name (as set in the configuration file) to be ignored (#1122)

How to upgrade

Run

docker pull obolibrary/odkfull:latest

to get the latest version of the image.

As for other point releases, there is no need to update your ODK-managed repositories with make update_repo.

ODK 1.5.3 (October 2024)

This is a point release of the Ontology Development Kit.

Changes since version 1.5.2

Software updates

• Base image updated to Ubuntu 24.04
• Python updated to version 3.12
• Several Python packages updated, including
  • Ontology Access Kit (oaklib 0.6.18)
  • LinkML (linkml 1.8.5 and linkml-runtime 1.8.3)
  • SSSOM (sssom-schema 1.0.0 and sssom 0.4.12)
• KGCL plugin for ROBOT updated to version 0.5.0
• SSSOM command-line tool and ROBOT plugin updated to version 0.9.0

Bugfixes

• Fix permission issue on macOS when seeding (#1105)
• Fix permission on the SSH socket (#1096)
• Fix lack of robustness of the odk.py script (#1097)

BREAKING CHANGE

J2cli, which is no longer maintained, has been removed and replaced with Jinjanator. Standard ODK-generated workflows never used that tool, but if you have custom workflows that are using the j2 command, you will need to update them to make them use jinjanate instead.

How to upgrade

Run

docker pull obolibrary/odkfull:latest

to get the latest version of the image.

As for other point releases, there is no need to update your ODK-managed repositories with make update_repo.

ODK 1.5.2 (July 2024)

Installation

Specifically 1.5.2:

docker pull obolibrary/odkfull:v1.5.2

But better just keep installing the latest version:

docker pull obolibrary/odkfull:latest

What's Changed

• Several Python packages updated, including:
  • oaklib (OAK) 0.6.11 (including some bugfixes to KGCL diffing and OBO format parsing)
  • sssom 0.4.11 and sssom-schema 0.15.2
  • linkml 1.8.1

Full Changelog: v1.5...v1.5.2

ODK 1.5.1 (May 2024)

What's Changed

• ROBOT updated to version 1.9.6. ROBOT has had some significant improvements, including a newer version of the Elk reasoner and finally (!) prefix map support for OBO format!
• KGCL plugin for ROBOT updated to version 0.4.0
• SSSOM command-line tool and plugin for ROBOT updated to version 0.7.7
• Several Python packages updated, including:
  • oaklib (OAK) 0.6.6
  • sssom 0.4.9 and sssom-schema 0.15.2
• Hotfixes:
  • Added back class-count-by-prefix.sparql (#1030)
  • Disabled table-reader mkdocs plugin (#1028)

Full Changelog: v1.5...v1.5.1

ODK 1.5 (March 2024)

We thank the community for all your support and feedback! Without your issues and contributions we would not be able to do our work!

Update instructions

https://oboacademy.github.io/obook/howto/odk-update/

Highlights and possibly breaking changes (Read this!)

• All processes within the ODK container now runs under the identity of an (unprivileged) user by default. This fixes the issue of generated files being owned by the superuser, when the Docker daemon itself runs as the superuser (as is the case by default on most GNU/Linux systems). See PR1, PR2, PR3.
  • Consequences: Some workflows that require superuser rights may not work anymore as expected.
  • Mitigation: If you have a workflow that requires being run as a superuser (for example, if you need to install extra Debian/Ubuntu packages via apt-get), set the environment variable ODK_USER_ID to 0 when running that workflow or, to always run all workflows as a superuser (as was the case in previous ODK versions), set the configuration parameter run_as_root to True before updating your repository.
• Change the definition of "base release". See PR.
  • Consequence: The base file now does not only contain the editors axioms in their raw form, but the axioms "as intended by the ontology developer", for example, including inferences. For the base-specification see here.
  • Mitigation: If you want a release that corresponds exactly to the old base file, use baselite instead.
• Allow refreshing the mirrors (externally downloaded ontologies) under IMP=false. See PR.
  • Consequence: you now cannot rely on IMP=false anymore if you want to avoid refreshing mirrors as well - you need to use IMP=false MIR=false instead!
• Make qc.yml ODK managed by default, so it is actually being updated along with the rest of the files. See PR.
  • Consequence: If you have overwritten that workflow with custom content, you have to follow the mitigation strategies.
  • Mitigation: (1) migrate the custom content to a differently named workflow or (2) deactivate syncning of that workflow manually by adding a workflows section to your ODK config.
• Generate custom_reports during release process PR.
  • Consequence: This means that all reports configured in ODK are automatically updated at release time, so you have more files need to review during a release.
  • Mitigation: Add your reports to .gitignore or remove them from your ODK config.
• We added the "International Edition" as a release product, including an entire workflow system for managing translations using the Babelon format. This feature is still under development, but works - feel free to reach out if you like to test it.

New and updated tooling

• New ROBOT Version 1.9.5 was added in a previous minor release.
• A lot of updated python tools, including OAK (0.5.25), SSSOM tools (0.4.4), LinkML (1.7.5), kgx (2.2.5) and curies (0.7.7). A full list of all available python tools and their versions is available here.
• The Babelon Toolkit (0.2.7) for managing multilingual ontologies has been added as well.
• Added table-reader plugin for mkdocs to directly integrate CSV files into your ontology documentation, see PR.
• Added Perl module Business::ISBN, see PR.
• Updated Apache Jena (4.9.0), Soufflé, Fastobo-validator, Ammonite REPL & Script-Runner (2.5.9), DOSDP-Tools (0.19.3), Relation-Graph (2.3.2).
• Added SSSOM-Java (0.7.5) (PR) and KGCL-Java (0.3.2) (PR) to odkfull.

New configuration options

• Add option to include defined-by annotation in imports, see PR.
• Add option release_annotate_inferred_axioms, see PR, to enable the annotation of inferred axioms during release.

Makefile workflows

• Make it possible to collect per-command resource usage data, see PR. This allows us to see which goals consume how much memory, and how long they take, to identify performance bottlenecks.
• Enable support for custom ROBOT plugins in ODK, see PR
• Add a test_fast goal to allow running tests without refreshing dependencies PR.
• Re-integrate LightRDF RDF/XML validation, see PR1, PR2
• Add SPARQL Check to find uses of deprecated DC PR
• Add release diff action, see PR. This allows posting diffs automatically as a comment to a Pull Request.
• Add convenience check if customised ROBOT report config is out of date PR. This allows checking if you are missing out on some new ROBOT report checks!
• Add a nicer, more comprehensive way to understand the versions of the tools used in ODK, see PR.

Runner and Infrastructure

• Allow passing configuration options to the OWL API, see PR.
• Automatically check whether the repository needs to be updated, see PR.
• Forward the host SSH agent socket into the container, see PR.

Bugfixes

• Fix import to use OBOBASE in base-iri, not URIBASE, see PR.
• Do not create individual import modules when use_base_merging is enabled, see PR.
• Make docs workflow configurable, see PR.
• Update illegal-date-violation.sparql to accept xsd:dateTime, see PR.
• Update URL to show CI status badge correctly on repo README.md, see PR.

For more detailed overview of the changes see:

• Milestone definition
• All changes since the last major release

ODK 1.4.3 (September 2023 2nd Edition)

This is a point release of the Ontology Development Kit. It uses updated versions of several components, including:

• ROBOT v1.9.5
• Ontology Access Kit (oaklib) 0.5.20
• Python curies packages 0.6.4
• Python SSSOM toolkit 0.3.41

This release does not change anything to the ODK-managed workflows, so running sh run.sh make update_repo to adapt your repositories to this new release is not needed. Just make sure to use the latest image by running docker pull obolibrary/odkfull.

ODK 1.4.2 (September 2023)

This is a point release of the Ontology Development Kit that brings up-to-date versions of several components:

• Apache Jena updated to version 4.9.0
• Soufflé updated to version 2.4
• Fastobo-validator updated to version 0.4.6
• Ammonite updated to version 2.5.9
• Several Python packages updated, including:
  • oaklib 0.5.17
  • sssom 0.3.40
  • sssom-schema 0.15.0
  • linkml 1.5.7
• /!\ The bioregistry dependency is removed in favour of the curies package

To use this release, run docker pull obolibrary/odkfull on your machine.

Updating your individual ODK-managed repositories with make update_repo is not needed as this release does not bring any change to the standard workflows.

ODK 1.4.1 (May 2023)

This is a point release of the Ontology Development Kit that brings up-to-date versions of several components:

• ROBOT version 1.9.4;
• Ontology Access Kit version 1.5.6;
• Bioregistry version 0.9.15;
• LinkML version 1.5.2.

To use this release, run docker pull obolibrary/odkfull on your machine.

Updating your individual ODK-managed repositories with make update_repo is not needed as this release does not bring any change to the standard workflows.

Full Changelog: v1.4...v1.4.1

ODK 1.4 (February 2023)

Update instructions

https://oboacademy.github.io/obook/howto/odk-update/

Changes from ODK 1.3.1 to 1.4

A full record of all changes can be seen here.

New and updated tooling

• New ROBOT Version 1.9.3
• A lot of updated python tools, including OAK (0.1.71), SSSOM tools (0.3.22), LinkML (1.4.4) and curies (0.4.2).
• gh is now installed in ODK, which means that GitHub workflows can be run directly through ODK. For example, try out the new public_release workflow which automatically creates a GitHub release for you.
• A full list of all available python tools and there vesions is available here

New configuration options

• Making the uribase configurable. You can now set the URI base in your myont-odk.yaml file to something different from http://purl.obolibrary.org/obo/, which enables developers from outside OBO to use ODK. Note that there is no guarantee that you can export your ontology to the OBO file format of you customise your baseuri this way!
• import_component_format: You can now configure the format in which your imports and components are serialised. For example, rather than owl (RDFXML), which used to be the default, you can set this option to ofn. For a complete reference see https://robot.obolibrary.org/convert.
• mirror_type: You can define the type of the mirror for each import. Supported: base, custom, no_mirror.
• release_use_reasoner: If false, no reasoning is performed for generating release files. This is only relevant for building application ontologies, where all components are already fully classified.
• sparql_test_on: You can define the list of input files to run the custom SPARQL queries. Supported: edit, and any release artefact, e.g myont-base.owl (example).
• use_edit_file_imports: If TRUE, use whatever imports you have in the edit file to create the release (default). If FALSE, components (and imports) are merged into the release independent of whether they are mentioned in the edit file or not. This can help for example in cases where import modules are so large, they cannot be checked into version control.
• The ci option now takes gitlab-ci as a value, which sets your repo up with basic Continuous Integration Testing for GitLab.

Makefile workflows

• Adding new make reason_test command (pull, issue)
• Adding intermediate artefact $(TMPDIR)/$(ONT)-preprocess.owl to the release workflow which enable the centralisation of preprocessing in a single make step. Basically, rather than creating release artefacts directly from the editors file (e.g. bfo-edit.owl), we add intermediate step from which all releases are derived. This intermediate can then be customised by the user (pull, issue)
• Introduces an experimental new release called "base-plus", which includes the inferred and non-redundant classification of the ontology. This is highly experimental and may be removed in a future release of ODK. The new goal is much more rigorous in removing axioms from other ontologies as well. The old base file can now be exported as an "editors release" instead (pull, issue).
• Introduces the option of skipping the use of reasoner during the release process (important for application ontologies), (pull, issue)
• Introduces a new mode that enable the use of ODK entirely without owl:imports in the edit file (this is great in case we want to use the ODK workflows but not check in any files, imports or components, into version control, like huge application ontologies), (pull, issue).
• Adds a feature to directly support ROBOT templates in components (example). Templates need to be activates with the project-level use_templates: TRUE option, and can then be used to define components, like in the example.
• Adds an option to do a public_release on Github which creates a GitHub release, tags it, and uploads release artefacts.
• Adds a release_diff workflow which creates a simple markdown report between the latest release and the current one.
• Adds a feature to directly support SSSOM mapping files, both in components and as standalone (example). This allows to curate, for example, SSSOM mapping files in tables and them release them as part of the general release process. It also allows extracting mappings from inside the Ontology automatically using the SSSOM toolkit.
• Changes in the src/ontology/run.sh wrapper script:
  • It is now possible to execute the Docker image through Singularity.
  • The IMAGE variable, which can used to specify an alternative ODK image, has been renamed to ODK_IMAGE.
  • A new variable ODK_TAG has been introduced, allowing to specify an alternative tag (default is latest). A tag may also be specified directly as part of the ODK_IMAGE variable (as in ODK_IMAGE=odkfull:v1.3.1).
  • A new variable ODK_BINDS has been introduced, allowing to specify extra bindings between a directory on the host computer and a directory inside the Docker container.
  • Variables used by the run.sh script can now be set in a src/ontology/run.sh.conf file, which will be sourced by the wrapper script.
  • With the config file option use_env_file_docker, if true, you can pass your local environment to the docker container. It is strongly advised to add run.sh.env to .gitignore before using this feature. Committing your environment to git may reveal passcodes and other private information!

The GitHub issues dealt with as part of this release

• Fix redundant filter in sparql queries by @anitacaron in #633
• Add docker requirement to top-level README by @kltm in #635
• Update to latest relation-graph by @balhoff in #634
• Make run.sh Java Option handling a bit nicer by @matentzn in #616
• Enable use of singularity as an option by @pcm32 in #640
• Allow to only push dev-tagged images to Docker Hub. by @gouttegd in #653
• Schema config documentation by @hkir-dev in #656
• Rename dosdp-tools directory. by @gouttegd in #652
• Rename IMAGE to ODK_IMAGE in the run.sh script. by @gouttegd in #655
• Build odkfull:dev from odklite:dev. by @gouttegd in #661
• Update constraints from within the odkbuild image by @gouttegd in #663
• Add constraints.txt make goal by @matentzn in #662
• Adding a preprocessing intermediate file to the ODK build process by @matentzn in #639
• Make sure the Python package six is present in odklite. by @gouttegd in #666
• Make Base IRI configurable by @matentzn in #617
• Update constraints.txt by @github-actions in #664
• Update catalog-v001.xml.jinja2 to reflect base IRI changes (bugfix) by @matentzn in #668
• Fix wrong config for recreating components by @matentzn in #670
• fixed parentheses in inject queries by @ehartley in #672
• Update constraints.txt by @github-actions in #667
• Update constraints.txt by @github-actions in #673
• Allow extra volume bindings and custom wrapper config file by @gouttegd in #678
• Update constraints.txt by @github-actions in #681
• Add documentation about custom sparql checks by @anitacaron in https://github.com/INCATools/ont...