Improve deployment dockerfiles

Summary

Improve the Cloud Run and Dataflow deployment dockerfiles and enable poetry-based packages on Dataflow.

Contents (#554)

Enhancements

• Use windpioneers/gdal-python as base in Cloud Run default deployment Dockerfile
• Use latest apache-beam base for Dataflow default deployment dockerfile
• In both the Cloud Run and Dataflow default deployment dockerfiles:
  • Improve dependency caching
  • Remove unnecessary installation of curl
  • Only install from one type of dependency file while building
  • Use /workspace as WORKDIR in Dockerfile
  • Use /root/.poetry for POETRY_HOME

Fixes

• Always install poetry in deployment dockerfiles

Operations

• Use latest octue/check-semantic-version GitHub action

Add code scanning

Summary

Enable automated code security scanning via GitHub's CodeQL.

Contents (#553)

Operations

• Add codeql.yml workflow to enable code scanning

Fix multi-threaded messaging and add detached service starting

Summary

Add a lock to avoid a race condition on message numbering when a child sets up a periodic monitor message thread to communicate with a parent. Detached serving of services in python has also been added.

Contents (#552)

New features

• Add ability to detach from locally serving services in Service.serve

Enhancements

• Allow multiple periodic monitor messages to be set up in an analysis
• Log when recording question UUID to delivered set in Cloud Run
• Log when the periodic monitor message threads are stopped
• Log that messages have been sent only after the send has been carried out successfully

Fixes

• Add threading lock for sending Pub/Sub messages and accessing/mutating the message number

Update app creation docs

Contents (#551)

Documentation

• Move app.py documentation to new page and update it with monitor message and analysis finalisation information

Dependencies

• Update lock file

Add periodic monitor messages and support poetry-based apps on Cloud Run

Summary

Improve the creation experience of Octue services.

Contents (#548)

New features

• Add ability to set up periodic monitor messages from Analysis
• Add poetry installation option to Cloud Run Dockerfile

Enhancements

• Allow revision tag to be set in octue deploy commands
• Tell user SRUID of service being deployed by octue deploy
• Change default internal service name to local/local:local
• Allow Cloud Run execution environment to be set by deployer

Fixes

• Ensure service revision tag is set as environment variable in deployers

Dependencies

• Update flake8 used for pre-commit checks
• Update lockfile to address certificate removal in certifi
• Use new poetry format for dev dependencies in pyproject.toml

Testing

• Update tox.ini to new format

Fix manifest upload in crash diagnostics

Contents (#545)

Fixes

• Avoid double-JSON-encoding input manifests in Service.ask

Dependencies

• Update lock file

Fix and improve crash diagnostics

Summary

This release fixes and adds new features to service crash diagnostics, making it much easier to locally reconstruct errors in a service. This especially true if it has children - crash diagnostics now include a child emulator for every question asked to a child by the service before it failed.

Contents (#541)

IMPORTANT: There are 3 breaking changes.

New features

• Add ability to load crash diagnostics data as a test fixture
• Record messages from each of a service's children when it crashes

Enhancements

• 💥 BREAKING CHANGE: Avoid downloading datasets by default in get-crash-diagnostics
• 💥 BREAKING CHANGE: Save recorded messages to attribute instead of file
• Log a warning if trying to download files from empty directory
• Add Serialisable.from_file method
• Add Manifest.update_dataset_paths method
• Add representation for Runner
• Avoid uploading null strings when values or manifests not present
• Add option to include .octue files when instantiating datasets
• Add ability to filter the files downloaded in GoogleCloudStorageClient.download_all_files

Fixes

• Save original unmutated input and configuration data for crash diagnostics
• Update manifests with local paths if datasets downloaded locally
• Ensure crash diagnostics upload works without manifests
• Ensure empty values or manifests are uploaded for crash diagnostics
• Fix splitting bucket name from signed cloud storage URLs

Refactoring

• 💥 BREAKING CHANGE: Rename filename parameter to path in Serialisable.to_file
• Factor out child instantiation in Runner into a method
• Move AnalysisLogHandlerSwitcher into log_handlers module
• Move AppFrom into new app_loading module
• Remove unused Service._sent_messages attribute
• Rename recorded_messages attributes to received_messages properties
• Factor out manifest/dataset and values uploading for crash diagnostics
• Reduce nesting in Dataset._instantiate_from_cloud
• Factor out app loading and running in Runner

Dependencies

• Update poetry.lock to avoid usage of insecure py package by pytest

Testing

• Make testing downloading crash diagnostics more efficient
• Test crash diagnostics with service that has its own children
• Use test fixture to speed up octue get-crash-diagnostics tests
• Test that dataset/file metadata is uploaded with crash diagnostics
• Test metadata is preserved with octue get-crash-diagnostics

Other

• Clarify service troubleshooting doc

---

Upgrade instructions

💥 Avoid downloading datasets by default in `get-crash-diagnostics`

To keep the previous behaviour, add the --download-datasets flag when using the octue get-crash-diagnostics CLI command

💥 Save recorded messages to attribute instead of file

Instead of using the record_messages_to parameter, set the record_messages parameter to True when using Child.ask or Service.wait_for_answer. Retrieve messages from the received_messages property of either class instance.

💥 Rename filename parameter to path in Serialisable.to_file

If using the positional argument filename in the to_file method of an instance of a Serialisable mixed-in class as a keyword argument, replace it with path.

Fix manifest upload and analysis error logging

Contents (#536)

Fixes

• Always use OctueJSONEncoder when encoding JSON
• Log analysis errors before attempting to save crash diagnostics in Runner

Operations

• Update author email address

Refactoring

• Use Manifest.serialise instead of Manifest.to_primitive in Manifest.to_cloud

Remove `mkver.conf` file and update troubleshooting docs

Contents (#533)

Operations

• Use new version of octue/check-semantic-version action and remove mkver.conf file
• Use latest octue/generate-pull-request-description action
• Add prettier to pre-commit checks

Other

• Fix octue.yaml code block in docs
• Improve troubleshooting services doc
• Update service ID usage in docs

Fully support service namespacing and revision tagging

Summary

This release add support and requirement for service namespaces and service revision tags. This allows services to be versioned and provides a more familiar format for service IDs that's, for example, similar to Docker image names (e.g. octue/example-service:latest). Full validation of service IDs comes as part of this.

The new octue deploy create-push-subscription has also been added as requested by users and will be accompanied by a GitHub action in another repository.

Contents (#529)

IMPORTANT: There are 4 breaking changes.

New features

• 💥 BREAKING CHANGE: Add support for and require revision tags for services
• 💥 BREAKING CHANGE: Validate service IDs before creating or questioning services
• Add ability to set service namespace, name, and revision tag by environment variable
• Add octue deploy create-push-subscription CLI command

Enhancements

• 💥 BREAKING CHANGE: Require services to have a namespace and replace use of organisation with namespace in service configurations
• 💥 BREAKING CHANGE: Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths
• Remove application of octue.services Pub/Sub namespace from Service
• Use a coolname revision tag to allow a service revision with a non-unique SRUID to start as a unique service revision when using the octue start CLI command
• Set the Service.id attribute to the user-friendly SRUID instead of the Pub/Sub-friendly SRUID
• Return SRUID from DataflowDeployer.deploy

Fixes

• Handle inability to find/load version compatibility data file
• Use latest Apache Beam base image in Dataflow Dockerfile
• Fix some log and error messages

Operations

• Speed up and simplify release workflow
• Use new Octue GitHub actions in workflows
• Improve installation process in main Dockerfile
• Use latest commit message checker

Dependencies

• Update Dataflow setup.py file

Refactoring

• Rename OCTUE_NAMESPACE to OCTUE_SERVICES_NAMESPACE
• Remove cool-naming of services when instantiated without id and simplify the name argument

---

Upgrade instructions

💥 Add support for and require revision tags for services

The ServiceConfiguration.service_id attribute and the SERVICE_ID environment variable have been removed.

• ServiceConfiguration: Use octue.cloud.service_id.create_service_id to create IDs from ServiceConfiguration.namespace and ServiceConfiguration.name

• SERVICE_ID environment variable: Use the new OCTUE_SERVICE_NAMESPACE, OCTUE_SERVICE_NAME, and OCTUE_SERVICE_REVISION_TAG environment variables.

💥 Validate service IDs before creating or questioning services

Update your services' namespaces, names, and revision tags to follow the requirements set out in the Creating services doc.

💥 Require services to have a namespace and replace use of organisation with namespace in service configurations

Provide the namespace key in the service configuration (the octue.yaml file). If you were using the organisation key before, the namespace key now replaces it. Examples of a namespace are your organisation's name or your GitHub username.

💥 Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths

The octue.services namespace is now mandatory and applied automatically in topic and subscription paths. Please stop providing the namespace argument if you were providing it before.