merged develop

Author: grallewellyn

CHANGELOG.md

@@ -5,6 +5,102 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+# [Unity Release 25.1] - 2025-04-08
+
+- SPS Version 2.6.0 (new)
+- OGC API Version 2.0.0 (unchanged)
+- OGC Python Client Version 2.0.1 (unchanged)
+
+## Overview
+
+This release contains several performance improvements to increase the scalability of the SPS system, including:
+- New EC2 types available when using the CWL DAG or CWL Modular DAG, including EC2 types with attached SSD storage.
+- Prioritization of downstream tasks of running DAGs with respect to upstream tasks of queued DAGs.
+- Removal access to shared EFS partition
+- Using EC2 types with increased CPU and memory for the core EKS node and core Airflow Pods
+
+Also, the Airflow database is backed up every day, and restored when the SPS system is redeployed.
+
+Additionally, this release includes much expanded support for the SRL (Sample Return Lander) use case,
+demonstrating triggering and chaining of data procesisng workflows for generation pof EDR and RDR products
+and image format conversion.
+
+Finally, all documentation has been revised and updated.
+
+## Caveats
+- Unless ports are specifically opened, this SPS installation is only accessible
+  through the top-level MDPS proxies. Users will need to login first with their
+  Cognito credentials, then with their Airflow credentials. This issue will be fixed
+  when the Airflow/Cognito integration is complete.
+- For the same reason, unless ports are specifically opened, the Airflow and OGC
+  APIs will not be accessible.
+
+## Upgrade Guide
+
+- This release is backward compatible with the previous version: it includes additional functionality and
+  performance improvements, but there are no API changes.
+- Because some configuration parameters have changed, the .tfvars files used to deploy EKS, Karpenter and Airflow via Terraform
+  should be re-generated to pick up the new values.
+
+## Repositories
+
+- unity-sps: <https://github.com/unity-sds/unity-sps/releases/tag/2.6.0>
+- unity-sps-ogc-processes-api: <https://github.com/unity-sds/unity-sps-ogc-processes-api/releases/tag/2.0.0>
+- unity-sps-ogc-processes-api-client-python: <https://github.com/unity-sds/unity-sps-ogc-processes-api-client-python/releases/tag/2.0.1>
+
+## Epics
+
+- EPIC: SPS Infrastructure
+  - [[Enhancement]: Update the Management Console to use the latest SDS version](https://github.com/unity-sds/unity-sps/issues/280)
+  - [[Bug]: Fix placement of Pods over Nodes](https://github.com/unity-sds/unity-sps/issues/304)
+  - [[Enhancement] Remove access to shared EFS partition in the CWL DAG](https://github.com/unity-sds/unity-sps/issues/318)
+  - [[Bug]: Update CPU and memory limits for worker nodes](https://github.com/unity-sds/unity-sps/issues/330)
+  - [[New Feature]: Experiment with using EC2 types with attached storage for better performance](https://github.com/unity-sds/unity-sps/issues/339)
+  - [[New Feature]: Log the EC2 Instance Id](https://github.com/unity-sds/unity-sps/issues/357)
+  - [[New Feature]: Scale up the SPS core resources](https://github.com/unity-sds/unity-sps/issues/358)
+  - [[New Feature]: Prioritize downstream tasks](https://github.com/unity-sds/unity-sps/issues/382)
+- EPIC: Deployment Enhancements
+  - [[New Feature]: Migrate database content during SPS upgrades](https://github.com/unity-sds/unity-sps/issues/253)
+- EPIC: Documentation Updates
+  - [[Documentation] OGC Documentation Enhancements](https://github.com/unity-sds/unity-sps/issues/274)
+  - [[Documentation]: Revise and update the OGC API Jupyter Notebook](https://github.com/unity-sds/unity-sps/issues/294)
+  - [[Documentation]: Provide a high level overview of the OGC Processes API](https://github.com/unity-sds/unity-sps/issues/295)
+  - [[Documentation]: Revise and update the OGC tutorial using CURL](https://github.com/unity-sds/unity-sps/issues/296)
+  - [[Documentation]: SPS overview for developers](https://github.com/unity-sds/unity-sps/issues/297)
+- EPIC: Continue Application Package Standardization
+  - [[New Feature]: Modularize the EMIT workflow](https://github.com/unity-sds/unity-sps/issues/302)
+- EPIC: TROPESS Support
+  - [[Enhancement]: Modular DAG Stage Out step too verbose](https://github.com/unity-sds/unity-sps/issues/307)
+- EPIC: Demonstrate Space Use Case
+  - [[Enhancement]Create SRL Initiator for EDRGen (.dat + .emd)](https://github.com/unity-sds/unity-sps/issues/287)
+  - [[New Feature]: Airflow DAG for EDRgen](https://github.com/unity-sds/unity-sps/issues/278)
+  - [[New Feature]: Airflow DAG for RDRgen](https://github.com/unity-sds/unity-sps/issues/279)
+  - [[New Feature]: Initiators for EDRgen processing](https://github.com/unity-sds/unity-sps/issues/283)
+  - [[New Feature]: Initiators for RDRgen processing](https://github.com/unity-sds/unity-sps/issues/284)
+  - [[New Feature]: Prototype routing via a DAG](https://github.com/unity-sds/unity-sps/issues/308)
+  - [[New Feature]: Investigate router DAG configuration via Airflow Configuration, SSM, S3 file](https://github.com/unity-sds/unity-sps/issues/324)
+
+## Docker Containers
+
+- ghcr.io/unity-sds/unity-sps/sps-airflow:2.6.0
+- ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.6.0
+- ghcr.io/unity-sds/unity-sps-ogc-processes-api/unity-sps-ogc-processes-api:2.0.0
+
+## Documentation
+
+- For Administrators:
+  - [SPS Deployment with Terraform](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/sps-deployment-with-terraform)
+  - [Interacting with an Existing SPS Deployment](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/interacting-with-an-existing-sps-deployment)
+  - [SPS Airflow Custom Docker Image Build Instructions](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/sps-airflow-custom-docker-image-build-instructions)
+  - [SPS Post Deployment Operations](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/sps-post-deployment-operations)
+  - [SPS Deployment with Marketplace](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/sps-deployment-with-marketplace)
+- For Deverlopers:
+  - [OGC Processes API Overview](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/developers-guide/ogc-processes-api-overview)
+  - [Tutorial: Using the OGC Processes API with CURL](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/developers-guide/tutorial-using-the-ogc-processes-api-with-curl)
+  - [Tutorial: Using the OGC processes API with Python](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/developers-guide/tutorial-using-the-ogc-processes-api-with-python)
+- For Users:
+  - [Tutorial: Register and Execute a CWL Workflow](https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/users-guide/tutorial-register-and-execute-a-cwl-workflow)
+
 # [Unity Release 24.4] - 2025-01-02
 
 ## Tags
@@ -41,6 +137,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Epics
 
+Release tickets: https://github.com/orgs/unity-sds/projects/3/views/53
+
 - EPIC: Airflow/WPS-T Integration
   - [[[New Feature]: Publish ogc-api-python-client to PyPi]](https://github.com/unity-sds/unity-sps/issues/225)
 - EPIC: `SPS Infrastructure`

airflow/dags/cwl_dag.py

@@ -149,10 +149,11 @@ def setup(ti=None, **context):
     logging.info(f"Selecting log level: {context['params']['log_level']}.")
 
 
-setup_task = PythonOperator(task_id="Setup", python_callable=setup, dag=dag)
+setup_task = PythonOperator(task_id="Setup", python_callable=setup, dag=dag, weight_rule="upstream")
 
 cwl_task = KubernetesPodOperator(
     retries=1,
+    weight_rule="upstream",
     task_id="cwl_task",
     namespace=POD_NAMESPACE,
     name="cwl-task-pod",
@@ -238,7 +239,11 @@ def cleanup(**context):
 
 
 cleanup_task = PythonOperator(
-    task_id="Cleanup", python_callable=cleanup, dag=dag, trigger_rule=TriggerRule.ALL_DONE
+    task_id="Cleanup",
+    python_callable=cleanup,
+    dag=dag,
+    trigger_rule=TriggerRule.ALL_DONE,
+    weight_rule="upstream",
 )
 
 chain(setup_task.as_setup(), cwl_task, cleanup_task.as_teardown(setups=setup_task))

airflow/dags/cwl_dag_modular.py

@@ -206,11 +206,12 @@ def setup(ti=None, **context):
     logging.info(f"Selecting log level: {context['params']['log_level']}.")
 
 
-setup_task = PythonOperator(task_id="Setup", python_callable=setup, dag=dag)
+setup_task = PythonOperator(task_id="Setup", python_callable=setup, dag=dag, weight_rule="upstream")
 
 
 cwl_task_processing = KubernetesPodOperator(
     retries=0,
+    weight_rule="upstream",
     task_id="cwl_task_processing",
     namespace=POD_NAMESPACE,
     name="cwl-task-pod",
@@ -288,7 +289,11 @@ def cleanup(**context):
 
 
 cleanup_task = PythonOperator(
-    task_id="Cleanup", python_callable=cleanup, dag=dag, trigger_rule=TriggerRule.ALL_DONE
+    task_id="Cleanup",
+    python_callable=cleanup,
+    dag=dag,
+    trigger_rule=TriggerRule.ALL_DONE,
+    weight_rule="upstream",
 )
 
 

airflow/helm/values.tmpl.yaml

@@ -353,7 +353,7 @@ extraEnv: |
   - name: AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC
     value: "1"
   - name: AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE
-    value: "8"
+    value: "16"
   - name: AIRFLOW__WEBSERVER__NAVBAR_COLOR
     value: "${webserver_navbar_color}"
   - name: AIRFLOW__WEBSERVER__INSTANCE_NAME
@@ -363,6 +363,6 @@ extraEnv: |
   - name: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
     value: "5"
   - name: AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT
-    value: "1000"
+    value: "1024"
   - name: AIRFLOW__WEBSERVER__EXPOSE_CONFIG
     value: "True"

airflow/plugins/unity_sps_utils.py

@@ -3,20 +3,17 @@
 """
 
 import os
-from datetime import datetime
 
 from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
 from kubernetes.client import models as k8s
 
 # Shared constants
 POD_NAMESPACE = "sps"  # The Kubernetes namespace within which the Pod is run (it must already exist)
-POD_LABEL = "cwl_task" + datetime.now().strftime(
-    "%Y%m%d_%H%M%S_%f"
-)  # unique pod label to assure each job runs on its own pod
+POD_LABEL = "cwl_task"
 
 # Note: each Pod is assigned the same label to assure that (via the anti-affinity requirements)
 # two Pods with the same label cannot run on the same Node
-SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.5.8"
+SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.5.10"
 
 NODE_POOL_DEFAULT = "airflow-kubernetes-pod-operator"
 NODE_POOL_HIGH_WORKLOAD = "airflow-kubernetes-pod-operator-high-workload"

pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "unity-sps"
-version = "2.4.0"
+version = "2.6.0"
 authors = [
-    { name = "Drew Meyers", email = "[email protected]" },
     { name = "Luca Cinquini", email = "[email protected]" },
-    { name = "Nikki Tebaldi", email = "[email protected]" },
-    { name = "Brad Lunsford", email = "[email protected]" }
+    { name = "Grace Llewellyn", email = "[email protected]" },
+    { name = "Brad Lunsford", email = "[email protected]" },
+    { name = "Nikki Tebaldi", email = "[email protected]" }
 ]
 description = "The science processing service area of Unity."
 classifiers = [

terraform-unity/README.md

@@ -188,10 +188,10 @@ terraform apply -no-color 2>&1 | tee apply_output.txt
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| <a name="input_airflow_docker_images"></a> [airflow\_docker\_images](#input\_airflow\_docker\_images) | Docker images for the associated Airflow services. | <pre>object({<br>    airflow = object({<br>      name = string<br>      tag  = string<br>    })<br>  })</pre> | <pre>{<br>  "airflow": {<br>    "name": "ghcr.io/unity-sds/unity-sps/sps-airflow",<br>    "tag": "2.5.8"<br>  }<br>}</pre> | no |
+| <a name="input_airflow_docker_images"></a> [airflow\_docker\_images](#input\_airflow\_docker\_images) | Docker images for the associated Airflow services. | <pre>object({<br>    airflow = object({<br>      name = string<br>      tag  = string<br>    })<br>  })</pre> | <pre>{<br>  "airflow": {<br>    "name": "ghcr.io/unity-sds/unity-sps/sps-airflow",<br>    "tag": "2.6.0"<br>  }<br>}</pre> | no |
 | <a name="input_airflow_webserver_password"></a> [airflow\_webserver\_password](#input\_airflow\_webserver\_password) | The password for the Airflow webserver and UI. | `string` | n/a | yes |
 | <a name="input_airflow_webserver_username"></a> [airflow\_webserver\_username](#input\_airflow\_webserver\_username) | The username for the Airflow webserver and UI. | `string` | `"admin"` | no |
-| <a name="input_dag_catalog_repo"></a> [dag\_catalog\_repo](#input\_dag\_catalog\_repo) | Git repository that stores the catalog of Airflow DAGs. | <pre>object({<br>    url                 = string<br>    ref                 = string<br>    dags_directory_path = string<br>  })</pre> | <pre>{<br>  "dags_directory_path": "airflow/dags",<br>  "ref": "develop",<br>  "url": "https://github.com/unity-sds/unity-sps.git"<br>}</pre> | no |
+| <a name="input_dag_catalog_repo"></a> [dag\_catalog\_repo](#input\_dag\_catalog\_repo) | Git repository that stores the catalog of Airflow DAGs. | <pre>object({<br>    url                 = string<br>    ref                 = string<br>    dags_directory_path = string<br>  })</pre> | <pre>{<br>  "dags_directory_path": "airflow/dags",<br>  "ref": "main",<br>  "url": "https://github.com/unity-sds/unity-sps.git"<br>}</pre> | no |
 | <a name="input_deployment_name"></a> [deployment\_name](#input\_deployment\_name) | The name of the deployment. | `string` | `""` | no |
 | <a name="input_helm_charts"></a> [helm\_charts](#input\_helm\_charts) | Helm charts for the associated services. | <pre>map(object({<br>    repository = string<br>    chart      = string<br>    version    = string<br>  }))</pre> | <pre>{<br>  "airflow": {<br>    "chart": "airflow",<br>    "repository": "https://airflow.apache.org",<br>    "version": "1.15.0"<br>  },<br>  "keda": {<br>    "chart": "keda",<br>    "repository": "https://kedacore.github.io/charts",<br>    "version": "v2.15.1"<br>  }<br>}</pre> | no |
 | <a name="input_installprefix"></a> [installprefix](#input\_installprefix) | The install prefix for the service area (unused) | `string` | `""` | no |
@@ -201,7 +201,7 @@ terraform apply -no-color 2>&1 | tee apply_output.txt
 | <a name="input_mcp_ami_owner_id"></a> [mcp\_ami\_owner\_id](#input\_mcp\_ami\_owner\_id) | The owner ID of the MCP AMIs | `string` | `"794625662971"` | no |
 | <a name="input_ogc_processes_docker_images"></a> [ogc\_processes\_docker\_images](#input\_ogc\_processes\_docker\_images) | Docker images for the associated OGC Processes API services. | <pre>object({<br>    ogc_processes_api = object({<br>      name = string<br>      tag  = string<br>    })<br>    git_sync = object({<br>      name = string<br>      tag  = string<br>    })<br>    redis = object({<br>      name = string<br>      tag  = string<br>    })<br>  })</pre> | <pre>{<br>  "git_sync": {<br>    "name": "registry.k8s.io/git-sync/git-sync",<br>    "tag": "v4.2.4"<br>  },<br>  "ogc_processes_api": {<br>    "name": "ghcr.io/unity-sds/unity-sps-ogc-processes-api/unity-sps-ogc-processes-api",<br>    "tag": "2.0.0"<br>  },<br>  "redis": {<br>    "name": "redis",<br>    "tag": "7.4.0"<br>  }<br>}</pre> | no |
 | <a name="input_project"></a> [project](#input\_project) | The project or mission deploying Unity SPS. | `string` | `"unity"` | no |
-| <a name="input_release"></a> [release](#input\_release) | The software release version. | `string` | `"24.4"` | no |
+| <a name="input_release"></a> [release](#input\_release) | The software release version. | `string` | `"25.1"` | no |
 | <a name="input_service_area"></a> [service\_area](#input\_service\_area) | The service area owner of the resources being deployed. | `string` | `"sps"` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Tags for the deployment (unused) | `map(string)` | <pre>{<br>  "empty": ""<br>}</pre> | no |
 | <a name="input_venue"></a> [venue](#input\_venue) | The MCP venue in which the resources will be deployed. | `string` | n/a | yes |