Address various issues with production docs (#747)

* doc prom storage

* doc ondemand certs

* decribe site groups and groups upgrade

* note extra image build needs config updating for new image

Author: sjpb

docs/openondemand.md

@@ -31,7 +31,13 @@ The above functionality is configured by running the `ansible/portal.yml` playbo
 See the [ansible/roles/openondemand/README.md](../ansible/roles/openondemand/README.md) for more details on the variables described below.
 
 The following variables have been given default values to allow Open OnDemand to work in a newly created environment without additional configuration, but generally should be overridden in `environments/site/inventory/group_vars/all/` with site-specific values:
-- `openondemand_servername` - this must be defined for both `openondemand` and `grafana` hosts (when Grafana is enabled). Default is `ansible_host` (i.e. the IP address) of the first host in the `openondemand` group.
+- `openondemand_servername` - this must be defined for both `openondemand` and
+  `grafana` hosts (when Grafana is enabled). The default is `ansible_host` (i.e.
+  the IP address) of the first host in the `openondemand` group. For production
+  environments this should probably be a DNS name.
+- `openondemand_ssl_cert` and `openondemand_ssl_cert_key` - by default a
+  self-signed certificate is generated, which should probably be replaced for
+  production environments.
 - `openondemand_auth` and any corresponding options. Defaults to `basic_pam`.
 - `openondemand_desktop_partition` and `openondemand_jupyter_partition` if the corresponding inventory groups are defined. Defaults to the first compute group defined in the `compute` OpenTofu variable in `environments/$ENV/tofu`.
 

docs/production.md

@@ -24,6 +24,10 @@ production-ready deployments.
     inventory = ../common/inventory,../site/inventory,inventory
     ```
 
+  In general only the `site` environment will need an `inventory/groups` file -
+  this is templated out by cookiecutter and should be modified as required to
+  enable features for all environments at the site.
+
 - To avoid divergence of configuration all possible overrides for group/role
 vars should be placed in `environments/site/inventory/group_vars/all/*.yml`
 unless the value really is environment-specific (e.g. DNS names for
@@ -127,7 +131,17 @@ and referenced from the `site` and `production` environments, e.g.:
   set the "attach" options and run `tofu apply` again - this should show there
   are no changes planned.
 
-- Configure Open OnDemand - see [specific documentation](openondemand.md).
+- Consider whether Prometheus storage configuration is required. By default:
+  - A 200GB state volume is provisioned (but see above)
+  - The common environment [sets](../environments/common/inventory/group_vars/all/prometheus.yml)
+    a maximum retention of 100 GB and 31 days
+  These may or may not be appropriate depending on the number of nodes, the
+  scrape interval, and other uses of the state volume (primarily the `slurmctld`
+  state and the `slurmdbd` database). See [docs/monitoring-and-logging](./monitoring-and-logging.md)
+  for more options.
+
+- Configure Open OnDemand - see [specific documentation](openondemand.md) which
+  notes specific variables required.
 
 - Remove the `demo_user` user from `environments/$ENV/inventory/group_vars/all/basic_users.yml`
 

docs/upgrades.md

@@ -50,6 +50,10 @@ All other commands should be run on the Ansible deploy host.
    site-specific configuration. In general changes to existing functionality will aim to be
    backward compatible. Alteration of site-specific configuration will usually only be
    necessary to use new functionality or where functionality has been upstreamed as above.
+   Note that the `environments/common/layouts/everything` file contains all possible
+   groups which can be used to enable features; diff this against your e.g.
+   `environments/site/inventory/groups` file to see new features which you may
+   wish to enable in the latter file.
 
    Make changes as necessary.
 
@@ -60,7 +64,9 @@ All other commands should be run on the Ansible deploy host.
 
     Note that some releases may not include new images. In this case use the image from the latest previous release with new images.
 
-1. If required, build an "extra" image with local modifications, see [docs/image-build.md](./image-build.md).
+1. If an "extra" image build with local modifications is required, update the
+   Packer build configuration to use the above new image and run a build. See
+   [docs/image-build.md](./image-build.md).
 
 1. Modify your site-specific environment to use this image, e.g. via `cluster_image_id` in `environments/$SITE_ENV/tofu/variables.tf`.