Automate backend functionality and model compatibility verification across Annif releases

[juhoinkinen]

Currently, verifying the correct working of backends and the compatibility of models trained on previous Annif versions is a manual process, typically performed by running the annif eval command by hand. This manual approach is time-consuming and error-prone, especially before new version releases.

This issue proposes automating the verification process to ensure:

• Automated execution of annif eval (and other relevant commands) for all supported backends and algorithms
• Systematic testing of model compatibility for models trained with previous Annif versions
• Integration of these automated checks into CI pipelines or as dedicated test scripts
• Documentation of the automated verification process for maintainers

Automating these steps will improve reliability, reduce manual effort, and help catch regressions or compatibility issues earlier in the release cycle.

[juhoinkinen]

Chatting with Copilot on agent mode gave this plan:

1. Model Compatibility Check (Backward Compatibility)

   Goal: Ensure models trained with the previous Annif version can be loaded and evaluated with the current development version, and that evaluation results do not change significantly.
   Process:
   Store models trained with the previous release in tests/compat_models/prev_version/.
   For each model:
   Load and evaluate using the current development version.
   Compare evaluation metrics to those obtained with the previous version (stored in a results file, e.g., prev_eval_results.json).
   Allow for a small, configurable threshold for metric differences.

2. Model Reproducibility Check (Forward Consistency)

   Goal: Ensure models trained with the current development version produce the same evaluation results as models trained with the previous version, when using the same data and configuration.
   Process:
   Train new models with the current development version using the same configs/data as before.
   Evaluate these new models.
   Compare their evaluation metrics to those from the previous version’s models (again, using prev_eval_results.json).
   Flag significant differences.

[juhoinkinen]

Continuing the chat and the initial implementation gave these ideas:

• the annif eval [OPTIONS] PROJECT_ID [PATHS] command has the option --metrics-file <FILENAME> to specify file in order to write evaluation metrics in JSON format, use that.
• for permanent storage it is best to use Hugging Face Hub:
  • utilize annif upload [OPTIONS] PROJECT_IDS_PATTERN REPO_ID and annif download [OPTIONS] PROJECT_IDS_PATTERN REPO_ID commands for models
  • utilize Hugging Face Model card for evaluation results, possibly via the eval_results attribute