centralized logging (#4)

* log unhandled exceptions so the contain request ID

* enable tracing in tests

* add terraform to deploy centralized log group

* add lock

* use centralized log group

- set invoke settings to 0 retries and 6 hour max event age
- fixed misspellings
- removed deployment_name variable

* don't use xray_recorder.capture as a decorator

* Update README.md

Author: pymonger

README.md

@@ -216,18 +216,56 @@ This guide provides a quick way to get started with our project. Please see our
 
 ### Setting Up the End-to-End Demo
 
-#### Deploying the Initiator
+#### Create the centralized log group
 
 1. Clone repo:
 
    ```
    git clone https://github.com/unity-sds/unity-initiator.git
    ```
 
+1. Change directory to the location of the centralized log group terraform:
+
+   ```
+   cd unity-initiator/terraform-unity/centralized_log_group/
+   ```
+
+1. Set up environment variables for `project` (by default `uod`) and venue (by default `dev`):
+
+   ```
+   export PROJECT=<your project, e.g. uod>
+   export VENUE=<your venue, e.g. dev>
+   ```
+
+1. Initialize terraform:
+
+   ```
+   terraform init
+   ```
+
+1. Run terraform apply:
+
+   ```
+   terraform apply \
+     --var project=${PROJECT} \
+     --var venue=${VENUE} \
+     -auto-approve
+   ```
+
+   **Take note of the `centralized_log_group_name` that is output by terraform. It will be used when setting up other resources (e.g. initiator, trigger and evaluator lambdas).**
+
+1. Export the `centralized_log_group_name` that was output from the centralized log group terraform deployment:
+
+   ```
+   export CENTRALIZED_LOG_GROUP=<your log group name, e.g. /unity/log/uod-dev-initiator-centralized-log-group>
+   ```
+
+#### Deploying the Initiator
+
 1. Change directory to the location of the inititator terraform:
 
    ```
-   cd unity-initiator/terraform-unity/initiator/
+   cd ../initiator/
    ```
 
 1. You will need an S3 bucket for terraform to stage the router Lambda zip file and router configuration YAML file during deployment. Create one or reuse an existing one and set an environment variable for it:
@@ -248,12 +286,6 @@ This guide provides a quick way to get started with our project. Please see our
    aws s3 cp test_router.yaml s3://${CODE_BUCKET}/test_router.yaml
    ```
 
-1. Set a project name:
-
-   ```
-   export PROJECT=gmanipon-test
-   ```
-
 1. Initialize terraform:
 
    ```
@@ -265,13 +297,20 @@ This guide provides a quick way to get started with our project. Please see our
    ```
    terraform apply \
      --var project=${PROJECT} \
+     --var venue=${VENUE} \
      --var code_bucket=${CODE_BUCKET} \
      --var router_config=s3://${CODE_BUCKET}/test_router.yaml \
      -auto-approve
    ```
 
    **Take note of the `initiator_topic_arn` that is output by terraform. It will be used when setting up any triggers.**
 
+1. Export the `initiator_topic_arn` that was output from the initiator terraform deployment:
+
+   ```
+   export INITIATOR_TOPIC_ARN=<initiator topic ARN>
+   ```
+
 #### Deploying Example Evaluators (SNS topic->SQS queue->Lambda)
 
 In this demo we will deploy 2 evaluators:
@@ -320,6 +359,8 @@ In this demo we will deploy 2 evaluators:
 
    ```
    terraform apply \
+     --var project=${PROJECT} \
+     --var venue=${VENUE} \
      --var evaluator_name=${EVALUATOR_NAME} \
      --var code_bucket=${CODE_BUCKET} \
      -auto-approve
@@ -361,6 +402,8 @@ In this demo we will deploy 2 evaluators:
 1. Run terraform apply:
    ```
    terraform apply \
+     --var project=${PROJECT} \
+     --var venue=${VENUE} \
      --var evaluator_name=${EVALUATOR_NAME} \
      --var code_bucket=${CODE_BUCKET} \
      -auto-approve
@@ -448,11 +491,12 @@ In this demo we will deploy 2 evaluators:
    terraform init
    ```
 
-1. Run terraform apply. Note the PROJECT and INITIATOR_TOPIC_ARN environment variables should have been set in the previous steps. If not set them again:
+1. Run terraform apply. Note the PROJECT, VENUE and INITIATOR_TOPIC_ARN environment variables should have been set in the previous steps. If not set them again:
 
    ```
    terraform apply \
      --var project=${PROJECT} \
+     --var venue=${VENUE} \
      --var initiator_topic_arn=${INITIATOR_TOPIC_ARN} \
      -auto-approve
    ```
@@ -505,6 +549,7 @@ In this demo we will deploy 2 evaluators:
    ```
    terraform apply \
      --var project=${PROJECT} \
+     --var venue=${VENUE} \
      --var code_bucket=${CODE_BUCKET} \
      --var initiator_topic_arn=${INITIATOR_TOPIC_ARN} \
      --var provider_id=${PROVIDER_ID} \

src/unity_initiator/cloud/lambda_handler.py

@@ -7,7 +7,7 @@
 from aws_xray_sdk.core import patch_all, xray_recorder
 
 from ..router import Router
-from ..utils.logger import logger
+from ..utils.logger import log_exceptions, logger
 
 # initialize the AWS X-Ray SDK
 patch_all()
@@ -16,33 +16,34 @@
 ROUTER = None
 
 
-@xray_recorder.capture("lambda_handler_base")
+@log_exceptions
 def lambda_handler_base(event, context):
     """Base lambda handler that instantiates a router, globally, and executes actions for a single payload."""
 
     logger.info("context: %s", context)
 
-    # TODO: Should use AppConfig. For now, either reading router config body in ROUTER_CFG env variable
-    # or from a url in ROUTER_CFG_URL env variable.
-    global ROUTER
-    if ROUTER is None:
-        router_cfg = os.environ.get("ROUTER_CFG", "").strip()
-        router_cfg_url = os.environ.get("ROUTER_CFG_URL", "").strip()
-        if router_cfg == "":
-            if router_cfg_url != "":
-                with smart_open.open(router_cfg_url, "r") as f:
-                    router_cfg = f.read()
-            else:
-                raise RuntimeError(
-                    "No router configuration specified via ROUTER_CFG or ROUTER_CFG_URL env variables."
-                )
-        fd, router_file = mkstemp(prefix="router_", suffix=".yaml", text=True)
-        with os.fdopen(fd, "w") as f:
-            f.write(router_cfg)
-        ROUTER = Router(router_file)
-        os.unlink(router_file)
-    xray_recorder.put_annotation("payload", event["payload"])
-    return ROUTER.execute_actions(event["payload"])
+    with xray_recorder.capture("lambda_handler_base"):
+        # TODO: Should use AppConfig. For now, either reading router config body in ROUTER_CFG env variable
+        # or from a url in ROUTER_CFG_URL env variable.
+        global ROUTER
+        if ROUTER is None:
+            router_cfg = os.environ.get("ROUTER_CFG", "").strip()
+            router_cfg_url = os.environ.get("ROUTER_CFG_URL", "").strip()
+            if router_cfg == "":
+                if router_cfg_url != "":
+                    with smart_open.open(router_cfg_url, "r") as f:
+                        router_cfg = f.read()
+                else:
+                    raise RuntimeError(
+                        "No router configuration specified via ROUTER_CFG or ROUTER_CFG_URL env variables."
+                    )
+            fd, router_file = mkstemp(prefix="router_", suffix=".yaml", text=True)
+            with os.fdopen(fd, "w") as f:
+                f.write(router_cfg)
+            ROUTER = Router(router_file)
+            os.unlink(router_file)
+        xray_recorder.put_annotation("payload", event["payload"])
+        return ROUTER.execute_actions(event["payload"])
 
 
 def lambda_handler_multiple_payloads(event, context):

src/unity_initiator/utils/logger.py

@@ -1,4 +1,6 @@
 import logging
+import traceback
+from functools import wraps
 
 # set logger and custom filter
 log_format = "[%(asctime)s: %(levelname)s/%(funcName)s] %(message)s"
@@ -16,3 +18,16 @@ def filter(self, record):
 logger = logging.getLogger("unity_initiator")
 logger.setLevel(logging.INFO)
 logger.addFilter(LogFilter())
+
+
+def log_exceptions(lambda_handler):
+    @wraps(lambda_handler)
+    def wrapper(event, context):
+        try:
+            return lambda_handler(event, context)
+        except Exception as err:
+            tb = traceback.format_exc()
+            logger.exception("Got exception: %s\n%s", str(err), tb)
+            raise
+
+    return wrapper

terraform-unity/centralized_log_group/.terraform.lock.hcl

@@ -0,0 +1,40 @@
+# terraform-unity
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | ~> 1.8.2 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >=5.50.0 |
+| <a name="requirement_local"></a> [local](#requirement\_local) | >=2.5.1 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | 5.65.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_cloudwatch_log_group.centralized_log_group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_project"></a> [project](#input\_project) | The unity project its installed into | `string` | `"uod"` | no |
+| <a name="input_venue"></a> [venue](#input\_venue) | The unity venue its installed into | `string` | `"dev"` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_centralized_log_group_name"></a> [centralized\_log\_group\_name](#output\_centralized\_log\_group\_name) | The name of the centralized log group |
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

terraform-unity/centralized_log_group/README.md

@@ -0,0 +1,15 @@
+locals {
+  log_group_name = "${var.project}-${var.venue}-initiator-centralized-log-group"
+  tags = {
+    Venue       = "dev"
+    ServiceArea = "cs"
+    Capability  = "initiator"
+    CapVersion  = "0.0.1"
+    Component   = "U-OD"
+    Name        = "${var.project}-${var.venue}-cs-initiator-od"
+    Proj        = var.project
+    CreatedBy   = "cs"
+    Env         = "dev"
+    Stack       = "U-OD"
+  }
+}

terraform-unity/centralized_log_group/locals.tf

@@ -0,0 +1,5 @@
+resource "aws_cloudwatch_log_group" "centralized_log_group" {
+  name              = "/unity/log/${local.log_group_name}"
+  retention_in_days = 14
+  tags              = local.tags
+}

terraform-unity/centralized_log_group/main.tf

@@ -0,0 +1,4 @@
+output "centralized_log_group_name" {
+  description = "The name of the centralized log group"
+  value       = aws_cloudwatch_log_group.centralized_log_group.name
+}

terraform-unity/centralized_log_group/output.tf

@@ -0,0 +1,11 @@
+variable "project" {
+  description = "The unity project its installed into"
+  type        = string
+  default     = "uod"
+}
+
+variable "venue" {
+  description = "The unity venue its installed into"
+  type        = string
+  default     = "dev"
+}

terraform-unity/centralized_log_group/variables.tf

@@ -0,0 +1,14 @@
+terraform {
+  required_version = "~> 1.8.2"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">=5.50.0"
+    }
+    local = {
+      source  = "hashicorp/local"
+      version = ">=2.5.1"
+    }
+  }
+}