Fix: Clamscan alarm fires frequently in dev and anvildev (#7441)

[achave11-ucsc]

Connected issues: #7441

Checklist

Author

• PR is assigned to the author
• PR is a draft
• Target branch is develop
• Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
• PR is linked to all issues it (partially) resolves
• PR description links to connected issues
• PR title matches¹ that of a linked issue _{or comment in PR explains why they're different}
• PR title references all linked issues
• For each linked issue, there is at least one commit whose title references that issue

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

• Added p tag to titles of partial commits
• This PR is labeled partial _{or completely resolves all linked issues}
• This PR partially resolves each of the linked issues _{or does not have the partial label}

Author (reindex)

• Added r tag to commit title _{or the changes introduced by this PR will not require reindexing of any deployment}
• This PR is labeled reindex:dev _{or the changes introduced by it will not require reindexing of dev}
• This PR is labeled reindex:anvildev _{or the changes introduced by it will not require reindexing of anvildev}
• This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
• This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
• This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod _{or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod}

Author (API changes)

• This PR and its linked issues are labeled API _{or this PR does not modify a REST API}
• Added a (A) tag to commit title for backwards (in)compatible changes _{or this PR does not modify a REST API}
• Updated REST API version number in app.py _{or this PR does not modify a REST API}

Author (upgrading deployments)

• Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (hotfixes)

• Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
• Reverted the temporary hotfixes for any linked issues _{or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues linked to this PR}

Author (before every review)

• Rebased PR branch on develop, squashed fixups from prior reviews
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile, Dockerfile or environment.boot}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}
• PR is awaiting requested review from a peer
• Status of PR is Review requested
• PR is assigned to only the peer

Peer reviewer (after approval)

Note that when requesting changes, the PR must be assigned back to the author.

• Actually approved the PR
• PR is not a draft
• PR is awaiting requested review from system administrator
• Status of PR is Review requested
• PR is assigned to only the system administrator

System administrator (after approval)

• Actually approved the PR
• Labeled linked issues as demo or no demo
• Commented on linked issues about demo expectations _{or all linked issues are labeled no demo}
• Decided if PR can be labeled no sandbox
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Status of PR is Approved
• PR is assigned to only the operator

Operator

• Checked reindex:… labels and r commit title tag
• Checked that demo expectations are clear _{or all linked issues are labeled no demo}
• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub

Operator (deploy .shared and .gitlab components)

• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of .gitlab component)

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (deploy runner image)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (sandbox build)

• Added sandbox label _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab dev _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab anvildev _{or PR is labeled no sandbox}
• Build passes in sandbox deployment _{or PR is labeled no sandbox}
• Build passes in anvilbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in sandbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in anvilbox deployment _{or PR is labeled no sandbox}
• Deleted unreferenced indices in sandbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in dev}
• Deleted unreferenced indices in anvilbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev}
• Started reindex in sandbox _{or this PR is not labeled reindex:dev}
• Started reindex in anvilbox _{or this PR is not labeled reindex:anvildev}
• Checked for failures in sandbox _{or this PR is not labeled reindex:dev}
• Checked for failures in anvilbox _{or this PR is not labeled reindex:anvildev}

Operator (merge the branch)

• All status checks passed and the PR is mergeable
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but only included p if the PR is also labeled partial}
• Pushed merge commit to GitHub
• Status of PR is Merged lower
• Status of blocked issues is Triage _{or no issues are blocked on the linked issues}

Operator (main build)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev
• Status of linked issues is Lower

Operator (reindex)

• Deindexed all unreferenced catalogs in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed all unreferenced catalogs in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Deindexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Indexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Indexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Started reindex in dev _{or this PR does not require reindexing dev}
• Started reindex in anvildev _{or this PR does not require reindexing anvildev}
• Checked for, triaged and possibly requeued messages in both fail queues in dev _{or this PR does not require reindexing dev}
• Checked for, triaged and possibly requeued messages in both fail queues in anvildev _{or this PR does not require reindexing anvildev}
• Emptied fail queues in dev _{or this PR does not require reindexing dev}
• Emptied fail queues in anvildev _{or this PR does not require reindexing anvildev}
• Restarted the Data Browser pipeline for the ucsc/hca/dev branch on GitLab in dev _{or this PR does not require reindexing dev}
• Restarted the Data Browser pipeline for the ucsc/lungmap/dev branch on GitLab in dev _{or this PR does not require reindexing dev}
• Restarted deploy_browser job in the GitLab pipeline for this PR in dev _{or this PR does not require reindexing dev}
• Restarted the Data Browser pipeline for the ucsc/anvil/anvildev branch on GitLab in anvildev _{or this PR does not require reindexing anvildev}
• Restarted deploy_browser job in the GitLab pipeline for this PR in anvildev _{or this PR does not require reindexing anvildev}

Operator (mirroring)

• Started mirroring in dev _{or this PR does not require mirroring dev}
• Started mirroring in anvildev _{or this PR does not require mirroring anvildev}
• Checked for, triaged and possibly requeued messages in mirror fail queue in dev _{or this PR does not require mirroring dev}
• Checked for, triaged and possibly requeued messages in mirror fail queue in anvildev _{or this PR does not require mirroring anvildev}
• Emptied mirror fail queue in dev _{or this PR does not require mirroring dev}
• Emptied mirror fail queue in anvildev _{or this PR does not require mirroring anvildev}

Operator

• Propagated the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.17%. Comparing base (3ceea95) to head (3f95648).
⚠️ Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #7444   +/-   ##
========================================
  Coverage    85.17%   85.17%           
========================================
  Files          155      155           
  Lines        22280    22280           
========================================
  Hits         18976    18976           
  Misses        3304     3304           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coveralls]

coverage: 85.365%. remained the same
when pulling 3f95648 on issues/achave11-ucsc/7441-bump-clamscan-alarm-period
into 3ceea95 on develop.

[achave11-ucsc]

The successful TF plan includes:

Changes to metric_alarm resources

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_cloudwatch_metric_alarm.clam_fail will be updated in-place
  ~ resource "aws_cloudwatch_metric_alarm" "clam_fail" {
        id                                    = "azul-clam_fail-dev.alarm"
      ~ period                                = 86400 -> 129600
        tags                                  = {
            "Name"                = "azul-clam_fail"
            "billing"             = "hca"
            "component"           = "azul-clam_fail"
            "deployment"          = "dev"
            "owner"               = "[email protected]"
            "service"             = "azul"
            "terraform_component" = "shared"
        }
        # (21 unchanged attributes hidden)
    }

  # aws_cloudwatch_metric_alarm.clamscan will be updated in-place
  ~ resource "aws_cloudwatch_metric_alarm" "clamscan" {
        id                                    = "azul-clamscan-dev.alarm"
        tags                                  = {
            "Name"                = "azul-clamscan"
            "billing"             = "hca"
            "component"           = "azul-clamscan"
            "deployment"          = "dev"
            "owner"               = "[email protected]"
            "service"             = "azul"
            "terraform_component" = "shared"
        }
        # (22 unchanged attributes hidden)

      - metric_query {
          - id          = "log_count_raw" -> null
          - period      = 0 -> null
          - return_data = false -> null
            # (3 unchanged attributes hidden)

          - metric {
              - dimensions  = {} -> null
              - metric_name = "azul-clamscan-dev" -> null
              - namespace   = "LogMetrics" -> null
              - period      = 86400 -> null
              - stat        = "Sum" -> null
                # (1 unchanged attribute hidden)
            }
        }
      - metric_query {
          - expression  = "FILL(log_count_raw, 0)" -> null
          - id          = "log_count_filled" -> null
          - period      = 0 -> null
          - return_data = true -> null
            # (2 unchanged attributes hidden)
        }
      + metric_query {
          + id          = "log_count_raw"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + metric_name = "azul-clamscan-dev"
              + namespace   = "LogMetrics"
              + period      = 129600
              + stat        = "Sum"
                # (1 unchanged attribute hidden)
            }
        }
      + metric_query {
          + expression  = "FILL(log_count_raw, 0)"
          + id          = "log_count_filled"
          + return_data = true
        }
    }

  # aws_cloudwatch_metric_alarm.freshclam will be updated in-place
  ~ resource "aws_cloudwatch_metric_alarm" "freshclam" {
        id                                    = "azul-freshclam-dev.alarm"
        tags                                  = {
            "Name"                = "azul-freshclam"
            "billing"             = "hca"
            "component"           = "azul-freshclam"
            "deployment"          = "dev"
            "owner"               = "[email protected]"
            "service"             = "azul"
            "terraform_component" = "shared"
        }
        # (22 unchanged attributes hidden)

      - metric_query {
          - id          = "log_count_raw" -> null
          - period      = 0 -> null
          - return_data = false -> null
            # (3 unchanged attributes hidden)

          - metric {
              - dimensions  = {} -> null
              - metric_name = "azul-freshclam-dev" -> null
              - namespace   = "LogMetrics" -> null
              - period      = 86400 -> null
              - stat        = "Sum" -> null
                # (1 unchanged attribute hidden)
            }
        }
      - metric_query {
          - expression  = "FILL(log_count_raw, 0)" -> null
          - id          = "log_count_filled" -> null
          - period      = 0 -> null
          - return_data = true -> null
            # (2 unchanged attributes hidden)
        }
      + metric_query {
          + id          = "log_count_raw"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + metric_name = "azul-freshclam-dev"
              + namespace   = "LogMetrics"
              + period      = 129600
              + stat        = "Sum"
                # (1 unchanged attribute hidden)
            }
        }
      + metric_query {
          + expression  = "FILL(log_count_raw, 0)"
          + id          = "log_count_filled"
          + return_data = true
        }
    }

[dsotirho-ucsc]

Approved.

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below