trigger workflow build

Author: mmguero

.trigger_workflow_build

@@ -1,2 +1,2 @@
 # this file exists solely for the purpose of being updated and seen by github to trigger a commit build action
-0
+1

docs/components.md

@@ -46,7 +46,7 @@ Malcolm leverages the following excellent open source tools, among others:
     * [reversinglabs/reversinglabs-yara-rules](https://github.com/reversinglabs/reversinglabs-yara-rules)
     * [SEKOIA-IO/Community](https://github.com/SEKOIA-IO/Community)
     * [volexity/threat-intel](https://github.com/volexity/threat-intel)
-* These [Zeek packages]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/shared/bin/zeek_install_plugins.sh):
+* These [Zeek packages]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/zeek/scripts/zeek_install_plugins.sh):
     * Amazon.com, Inc.'s [ICS protocol](https://github.com/amzn?q=zeek) analyzers
     * Andrew Klaus's [Sniffpass](https://github.com/cybera/zeek-sniffpass) plugin for detecting cleartext passwords in HTTP POST requests
     * Andrew Klaus's [zeek-httpattacks](https://github.com/precurse/zeek-httpattacks) plugin for detecting noncompliant HTTP requests

docs/contributing-github-runners.md

@@ -2,7 +2,7 @@
 
 Users who have [forked](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) Malcolm on [GitHub]({{ site.github.repository_url }}) can use GitHub-hosted [runners](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) to build Malcolm images, as well as [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs, and push those images to GitHub's [ghcr.io container registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry).
 
-The resulting images are named according to the pattern `ghcr.io/username/malcolm/image:branch` (e.g., if the user has forked Malcolm with the GitHub user `romeogdetlevjr`, the `Arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run a local instance of Malcolm using these images instead of the official `ghcr.io/idaholab` ones, users will need to edit their `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the Bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images.
+The resulting images are named according to the pattern `ghcr.io/username/malcolm/image:branch` (e.g., if the user has forked Malcolm with the GitHub user `romeogdetlevjr`, the `Arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run a local instance of Malcolm using these images instead of the official `ghcr.io/idaholab` ones, users will need to edit their `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the Bash helper script [`./scripts/github_image_helper.sh`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/scripts/github_image_helper.sh) to pull and re-tag the images.
 
 [Workflow files]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/.github/workflows/) are provided that contain instructions to build the images using GitHub-hosted runners, as well as [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. 
 

docs/contributing-local-modifications.md

@@ -648,4 +648,4 @@ Another method for modifying local copies of Malcolm's services' containers is t
 
 For example, imagine a user wanted to create a Malcolm container that includes a new dashboard for OpenSearch Dashboards and a new enrichment filter `.conf` file for Logstash. After placing these files under `./dashboards/dashboards` and `./logstash/pipelines/enrichment`, respectively, in the Malcolm working copy, run `./build.sh dashboards-helper logstash` to build just those containers. After the build completes, run `docker images` to see the fresh images for `ghcr.io/idaholab/malcolm/dashboards-helper` and `ghcr.io/idaholab/malcolm/logstash-oss`. Users may need to review the contents of the [Dockerfiles]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/Dockerfiles) to determine the correct service and filesystem location within that service's image depending on the nature of the task.
 
-Alternately, forks of Malcolm on GitHub contain [workflow files]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/.github/workflows/) that contain instructions for GitHub to build the images and [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolm/image:branch` (e.g., if the GitHub user `romeogdetlevjr` has forked Malcolm, the `arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run a local instance of Malcolm using these images instead of the official ones, users would need to edit their `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images.
+Alternately, forks of Malcolm on GitHub contain [workflow files]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/.github/workflows/) that contain instructions for GitHub to build the images and [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolm/image:branch` (e.g., if the GitHub user `romeogdetlevjr` has forked Malcolm, the `arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run a local instance of Malcolm using these images instead of the official ones, users would need to edit their `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script [`./scripts/github_image_helper.sh`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/scripts/github_image_helper.sh) to pull and re-tag the images.

docs/contributing-zeek.md

@@ -8,7 +8,7 @@ Other changes to Zeek's behavior could be made by modifying [local.zeek]({{ site
 
 ## <a name="ZeekPackage"></a>Adding a new Zeek package
 
-The easiest way to add a new Zeek package to Malcolm is to add the git URL of that package to the `ZKG_GITHUB_URLS` array in [zeek_install_plugins.sh]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/shared/bin/zeek_install_plugins.sh) script and then [rebuilding](development.md#Build) the `zeek` image. This will cause the package to be installed (via the [`zkg`](https://docs.zeek.org/projects/package-manager/en/stable/zkg.html) command-line tool). See [Parsing new Zeek logs](contributing-logstash.md#LogstashZeek) on how to process any new `.log` files if the package generates them.
+The easiest way to add a new Zeek package to Malcolm is to add the git URL of that package to the `ZKG_GITHUB_URLS` array in [zeek_install_plugins.sh]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/zeek/scripts/zeek_install_plugins.sh) script and then [rebuilding](development.md#Build) the `zeek` image. This will cause the package to be installed (via the [`zkg`](https://docs.zeek.org/projects/package-manager/en/stable/zkg.html) command-line tool). See [Parsing new Zeek logs](contributing-logstash.md#LogstashZeek) on how to process any new `.log` files if the package generates them.
 
 ## <a name="ContributingZeekIntel"></a>Zeek Intelligence Framework
 

docs/custom-rules.md

@@ -62,7 +62,7 @@ If the `SURICATA_CUSTOM_RULES_ONLY` [environment variable](malcolm-config.md#Mal
 
 ### Configuration
 
-Suricata uses the [YAML format for configuration](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html), and the main `suricata.yaml` file is generated by Malcolm [dynamically at runtime]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/shared/bin/suricata_config_populate.py).
+Suricata uses the [YAML format for configuration](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html), and the main `suricata.yaml` file is generated by Malcolm [dynamically at runtime]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/suricata/scripts/suricata_config_populate.py).
 
 The contents of the `suricata.yaml` file can be adjusted via [environment variables](malcolm-config.md#MalcolmConfigEnvVars) found in [`suricata.env`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/config/suricata.env.example).
 

docs/development.md

@@ -13,31 +13,35 @@ Checking out the [Malcolm source code]({{ site.github.repository_url }}/tree/{{
 * `dashboards` - code and configuration for the `dashboards` container for creating additional ad-hoc visualizations and dashboards beyond that which is provided by Arkime Viewer
 * `Dockerfiles` - a directory containing build instructions for Malcolm's images
 * `docs` - a directory containing instructions and documentation
+* `file-upload` - code and configuration for the `upload` container that serves a web browser-based upload form for uploading PCAP files and Zeek logs, and serves an SFTP share as an alternate upload
 * `filebeat` - code and configuration for the `filebeat` container that ingests Zeek logs and forwards them to the `logstash` container
+* `filescan-logs` - an initially empty directory into which file scanning result logs will be generated
 * `filescan` - code and configuration for the `filescan` container that can scan files extracted by Zeek
-* `file-upload` - code and configuration for the `upload` container that serves a web browser-based upload form for uploading PCAP files and Zeek logs, and serves an SFTP share as an alternate upload
 * `freq-server` - code and configuration for the `freq` container used for calculating entropy of strings
 * `hedgehog-raspi` - code and configuration for building a [Hedgehog Linux](live-analysis.md#Hedgehog) [Raspberry Pi image](hedgehog-raspi-build.md#HedgehogRaspiBuild)
 * `htadmin` - configuration for the `htadmin` user account management container
+* `keycloak` - code for the [`keycloak`](authsetup.md#AuthKeycloak) container
 * `logstash` - code and configuration for the `logstash` container that parses Zeek logs and forwards them to the `opensearch` container
 * `malcolm-iso` - code and configuration for building an [installer ISO](malcolm-iso.md#ISO) for a minimal Debian-based Linux installation for running Malcolm
 * `netbox` - code and configuration for the `netbox` container which provides asset management capabilities
 * `nginx` - configuration for the `nginx` reverse-proxy container
-* `opensearch` - an initially empty directory where the OpenSearch database instance will reside
 * `opensearch-backup` - an initially empty directory for storing OpenSearch [index snapshots](index-management.md#IndexManagement) 
-* `pcap` - an initially empty directory for PCAP files to be uploaded, processed, and stored
+* `opensearch-config` - code and scripts used during the initialization of the `opensearch` container
+* `opensearch` - an initially empty directory where the OpenSearch database instance will reside
 * `pcap-capture` - code and configuration for the `pcap-capture` container that can capture network traffic
 * `pcap-monitor` - code and configuration for the `pcap-monitor` container that watches for new or uploaded PCAP files and notifies the other services to process them
+* `pcap` - an initially empty directory for PCAP files to be uploaded, processed, and stored
 * `postgres-scripts` - code used in the `postgres` container for database initialization
+* `postgres` - an initially empty directory to hold the PostgreSQL database used by NetBox and Keycloak
 * `redis` - an initially empty directory where the Redis database runtime files will reside
 * `scripts` - control scripts for starting, stopping, restarting, etc., Malcolm
 * `shared` - miscellaneous code used by various Malcolm components 
-* `strelka` - code for the `strelka-` containers used in by `filescan` for file scanning
-* `suricata` - code and configuration for the `suricata` container that handles PCAP processing using Suricata
+* `strelka` - code and configuration for the `strelka-` containers used in by `filescan` for file scanning
 * `suricata-logs` - an initially empty directory for Suricata logs to be uploaded, processed, and stored
+* `suricata` - code and configuration for the `suricata` container that handles PCAP processing using Suricata
 * `yara` - a directory for custom user-provided YARA rules
-* `zeek` - code and configuration for the `Zeek` container that handles PCAP processing using Zeek
 * `zeek-logs` - an initially empty directory for Zeek logs to be uploaded, processed, and stored
+* `zeek` - code and configuration for the `Zeek` container that handles PCAP processing using Zeek
 * `_includes` and `_layouts` - templates for the HTML version of the documentation
 * `.github` - GitHub workflow actions used to build [Malcolm using GitHub runners](contributing-github-runners.md#GitHubRunners)
 
@@ -80,7 +84,7 @@ Then, go take a walk or something since it will be a while. When you are done, y
 * `ghcr.io/idaholab/malcolm/suricata` (based on `debian:13-slim`)
 * `ghcr.io/idaholab/malcolm/zeek` (based on `zeek/zeek`)
 
-Alternately, if you have forked Malcolm on GitHub, [workflow files]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/.github/workflows/) are provided that contain instructions for GitHub to build the images, as well as [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolm/image:branch` (e.g., if you have forked Malcolm with the GitHub user `romeogdetlevjr`, the `Arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you will need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images.
+Alternately, if you have forked Malcolm on GitHub, [workflow files]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/.github/workflows/) are provided that contain instructions for GitHub to build the images, as well as [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolm/image:branch` (e.g., if you have forked Malcolm with the GitHub user `romeogdetlevjr`, the `Arkime` container built for the `main` branch would be named `ghcr.io/romeogdetlevjr/malcolm/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you will need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script [`./scripts/github_image_helper.sh`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/scripts/github_image_helper.sh) to pull and re-tag the images.
 
 # <a name="Packager"></a>Pre-Packaged installation files
 

docs/live-analysis.md

@@ -133,7 +133,7 @@ Aside from the settings mentioned above, to quote the Arkime documentation, ofte
 
 Suricata's resource utilization and performance can be tuned using [environment variables](malcolm-config.md#MalcolmConfigEnvVars) that can be added or modified in [`suricata-live.env`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/config/suricata-live.env.example).
 
-Upon starting, Malcolm's [`suricata_config_populate.py`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/config/shared/bin/suricata_config_populate.py) script generates the `suricata.yaml` configuration file (see (see [`suricata.yaml.in`](https://github.com/OISF/suricata/blob/master/suricata.yaml.in) and the [Suricata documentation](https://suricata.readthedocs.io/en/latest/configuration/suricata-yaml.html)). The `suricata_config_populate.py` script can use **many** environment variables when generating `suricata.yaml`. See the `DEFAULT_VARS` array in the script for a full list. Note that the environment variables must be prefixed with `SURICATA_` when defined in `suricata-live.env`.
+Upon starting, Malcolm's [`suricata_config_populate.py`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/suricata/scripts/suricata_config_populate.py) script generates the `suricata.yaml` configuration file (see (see [`suricata.yaml.in`](https://github.com/OISF/suricata/blob/master/suricata.yaml.in) and the [Suricata documentation](https://suricata.readthedocs.io/en/latest/configuration/suricata-yaml.html)). The `suricata_config_populate.py` script can use **many** environment variables when generating `suricata.yaml`. See the `DEFAULT_VARS` array in the script for a full list. Note that the environment variables must be prefixed with `SURICATA_` when defined in `suricata-live.env`.
 
 The following environment variables related to tuning Suricata for live packet capture may be of particular interest, but this list is by no means exhaustive: