Merge pull request #36 from octue/release/0.1.5

Release/0.1.5 Enabling task broker and improvements to CLI options

Author: thclark

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,60 @@
+
+**I'm submitting a ...**
+  - [ ] support request
+  - [ ] bug report
+  - [ ] feature request
+  
+Please fill out the relevant sections below.
+
+---
+
+## About Me
+
+If you're new to the twined ecosystem, and you don't mind, it'd be great to hear a little bit of the wider context about
+you and what you're working on, so we can understand how best to help.
+
+
+
+## Support request
+
+- [ ] I've searched the list of existing issues
+- [ ] I expected the docs to cover it in [this section](please provide link to where you'd expect to find help on this in the docs) but they don't cover it.
+
+I'm trying to [describe what you're trying to do].
+
+
+
+## Feature request
+ 
+### Use Case
+
+Please [describe your motivation and use case].
+
+### Current state
+
+Please describe what you're doing presently to work around this or achieve what you're doing.
+
+
+ 
+## Bug report
+
+### What is the current behavior?
+
+*If the current behavior is a bug, please provide any stack tracesthe steps to reproduce and if possible a minimal demo of the problem*
+
+### What is the expected behavior?
+
+### Your environment
+  
+  - Library Version: x.y.z
+  - Platform MacOS | Windows | Linux | Other
+
+### Other information
+ 
+Please give as much detail as you can, like:
+
+- [ ] detailed explanation,
+- [ ] stacktraces
+- [ ] related issues
+- [ ] suggestions how to fix
+- [ ] links for us to have context, eg. stackoverflow, gitter, etc

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,39 @@
+
+<!-- PRs from release/x.y.z branches will be used to create Release Notes so make sure they contain everything -->
+<!-- Please don't add features not discussed in an issue first -->
+
+## Contents
+
+### New Features
+
+<!-- Write `Closes #xyz` to link issues that this PR completes,
+           `WIP on #xyz` to link issues that this PR works toward. -->
+
+- [ ] Closes #xyz New thing for doing stuff
+- [ ] Closes #abc Another new thing for doing even more stuff
+
+### Breaking changes
+
+<!-- Breaking changes and how to work around them should be covered in the docs -->
+
+- [ ] Breaks a thing
+- [ ] Breaks another thing
+
+### Minor fixes and improvements
+
+- [ ] Closes #pqr An annoying bug
+- [ ] Closes #stu A tweak
+
+
+## Quality Checklist
+
+- [ ] New features are fully tested (No matter how much Coverage Karma you have)
+- [ ] **[v0.2 onward]** New features are included in the documentation
+- [ ] **[v0.2 onward]** Breaking changes are documented with clear instructions of what 
+
+### Coverage Karma
+
+- [ ] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?
+
+*Coverage Karma can be earned by disproportionally increasing test coverage in the rest of your contributions.
+It's an honesty policy - you keep count of your own.  

.github/workflows/python-ci.yml

@@ -0,0 +1,67 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions.
+#
+# On successful test, the package will be published. For candidate releases, the package will be
+#  published to test.pypi.org server (to ensure the process works). For merges to master, the
+#  package will be published live.
+
+name: python-ci
+
+on: [push]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    env:
+      USING_COVERAGE: '3.8'
+    strategy:
+      matrix:
+        python: [3.6, 3.7, 3.8]
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install Tox and any other packages
+        run: pip install tox
+      - name: Run Tox
+        run: tox -e py
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v1
+        with:
+          file: coverage.xml
+          fail_ci_if_error: true
+          token: ${{ secrets.CODECOV_TOKEN }}
+
+  publish:
+    if: contains(github.ref, 'main') || contains(github.ref, 'release/')
+    runs-on: ubuntu-latest
+    needs: tests
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: Make package
+        run: |
+          python3 -m pip install --upgrade setuptools wheel
+          python3 setup.py sdist bdist_wheel
+      - name: Test package is publishable with PyPI test server
+        if: contains(github.ref, 'release/')
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          user: __token__
+          password: ${{ secrets.TEST_PYPI_TOKEN }}
+          repository_url: https://test.pypi.org/legacy/
+          skip_existing: true
+      - name: Publish latest package to PyPI
+        if: contains(github.ref, 'main')
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_TOKEN }}
+          verbose: true
+

.travis.yml

@@ -1,5 +1,4 @@
 [![PyPI version](https://badge.fury.io/py/octue.svg)](https://badge.fury.io/py/octue)
-[![Build Status](https://travis-ci.com/octue/octue-sdk-python.svg?branch=master)](https://travis-ci.com/octue/octue-sdk-python)
 [![codecov](https://codecov.io/gh/octue/octue-sdk-python/branch/master/graph/badge.svg?token=4KdR7fmwcT)](undefined)
 [![Documentation Status](https://readthedocs.org/projects/octue/badge/?version=latest)](https://octue.readthedocs.io/en/latest/?badge=latest)
 [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

README.md

@@ -1,5 +1,6 @@
 from .cli import octue_cli
-from .runner import LOG_FORMAT, Runner
+from .logging_handlers import LOG_FORMAT
+from .runner import Runner
 
 
 __all__ = "LOG_FORMAT", "octue_cli", "Runner"

octue/__init__.py

@@ -4,15 +4,20 @@
 import pkg_resources
 
 from octue.definitions import FOLDER_DEFAULTS, MANIFEST_FILENAME, VALUES_FILENAME
+from octue.logging_handlers import get_remote_handler
 from octue.runner import Runner
 
 
+global_cli_context = {}
+
+
 @click.group(context_settings={"help_option_names": ["-h", "--help"]})
 @click.option(
     "--id",
     default=None,
+    type=click.UUID,
     show_default=True,
-    help="Id of the analysis being undertaken. None (for local use) will cause a unique ID to be generated.",
+    help="UUID of the analysis being undertaken. None (for local use) will cause a unique ID to be generated.",
 )
 @click.option(
     "--skip-checks/--no-skip-checks",
@@ -22,6 +27,7 @@
     help="Skips the input checking. This can be a timesaver if you already checked "
     "data directories (especially if manifests are large).",
 )
+@click.option("--logger-uri", default=None, show_default=True, help="Stream logs to a websocket at the given URI.")
 @click.option(
     "--log-level",
     default="info",
@@ -37,16 +43,18 @@
     help="Forces a reset of analysis cache and outputs [For future use, currently not implemented]",
 )
 @click.version_option(version=pkg_resources.get_distribution("octue").version)
-@click.pass_context
-def octue_cli(ctx, id, skip_checks, log_level, force_reset):
+def octue_cli(id, skip_checks, logger_uri, log_level, force_reset):
     """ Octue CLI, enabling a data service / digital twin to be run like a command line application.
 
     When acting in CLI mode, results are read from and written to disk (see
     https://octue-python-sdk.readthedocs.io/en/latest/ for how to run your application directly without the CLI).
     Once your application has run, you'll be able to find output values and manifest in your specified --output-dir.
     """
-    # TODO Forward command line options to runner via ctx
-    ctx.ensure_object(dict)
+    global_cli_context["analysis_id"] = id
+    global_cli_context["skip_checks"] = skip_checks
+    global_cli_context["logger_uri"] = logger_uri
+    global_cli_context["log_level"] = log_level.upper()
+    global_cli_context["force_reset"] = force_reset
 
 
 @octue_cli.command()
@@ -86,9 +94,7 @@ def octue_cli(ctx, id, skip_checks, log_level, force_reset):
     show_default=True,
     help="Directory to write outputs as files (overrides --data-dir).",
 )
-@click.option(
-    "--twine", type=click.Path(), default="twine.json", show_default=True, help="Location of Twine file.",
-)
+@click.option("--twine", type=click.Path(), default="twine.json", show_default=True, help="Location of Twine file.")
 def run(app_dir, data_dir, config_dir, input_dir, output_dir, twine):
     config_dir = config_dir or os.path.join(data_dir, FOLDER_DEFAULTS["configuration"])
     input_dir = input_dir or os.path.join(data_dir, FOLDER_DEFAULTS["input"])
@@ -98,12 +104,24 @@ def run(app_dir, data_dir, config_dir, input_dir, output_dir, twine):
         twine=twine,
         configuration_values=os.path.join(config_dir, VALUES_FILENAME),
         configuration_manifest=os.path.join(config_dir, MANIFEST_FILENAME),
+        log_level=global_cli_context["log_level"],
     )
+
+    if global_cli_context["logger_uri"]:
+        handler = get_remote_handler(
+            logger_uri=global_cli_context["logger_uri"], log_level=global_cli_context["log_level"]
+        )
+    else:
+        handler = None
+
     analysis = runner.run(
         app_src=app_dir,
+        analysis_id=global_cli_context["analysis_id"],
+        handler=handler,
         input_values=os.path.join(input_dir, VALUES_FILENAME),
         input_manifest=os.path.join(input_dir, MANIFEST_FILENAME),
         output_manifest_path=os.path.join(output_dir, MANIFEST_FILENAME),
+        skip_checks=global_cli_context["skip_checks"],
     )
     analysis.finalise(output_dir=output_dir)
     return 0

octue/cli.py

@@ -0,0 +1,32 @@
+import logging
+import logging.handlers
+from urllib.parse import urlparse
+
+
+# Logging format for analysis runs. All handlers should use this logging format, to make logs consistently parseable
+LOG_FORMAT = "%(name)s %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s"
+
+
+def get_default_handler(log_level):
+    """ Gets a basic console handler set up for logging analyses
+    """
+    console_handler = logging.StreamHandler()
+    console_handler.setLevel(log_level)
+    formatter = logging.Formatter(LOG_FORMAT)
+    console_handler.setFormatter(formatter)
+    return console_handler
+
+
+def get_remote_handler(logger_uri, log_level):
+    """Get a log handler for streaming logs to a remote URI accessed via HTTP or HTTPS."""
+    parsed_uri = urlparse(logger_uri)
+
+    if parsed_uri.scheme not in {"ws", "wss"}:
+        raise ValueError(
+            f"Only WS and WSS protocols currently supported for remote logger URI. Received {logger_uri!r}."
+        )
+
+    handler = logging.handlers.SocketHandler(host=parsed_uri.hostname, port=parsed_uri.port)
+    handler.setLevel(log_level)
+    handler.setFormatter(logging.Formatter(LOG_FORMAT))
+    return handler

octue/logging_handlers.py

@@ -48,7 +48,7 @@ class Analysis(Identifiable, Loggable, Serialisable, Taggable):
     :parameter logger: Optional logging.Logger instance attached to the analysis
     """
 
-    def __init__(self, twine, **kwargs):
+    def __init__(self, twine, skip_checks=False, **kwargs):
         """ Constructor of Analysis instance
         """
 
@@ -57,6 +57,7 @@ def __init__(self, twine, **kwargs):
             twine = Twine(source=twine)
 
         self.twine = twine
+        self._skip_checks = skip_checks
 
         # Pop any possible strand data sources before init superclasses (and tie them to protected attributes)
         strand_kwargs = ((name, kwargs.pop(name, None)) for name in ALL_STRANDS)