Nginx: using cont-finish.d for more graceful shutdown (#67)

* Nginx: using cont-init.d for more graceful shutdown

First suppress initial TERM/HUP, then use 
cont-finish.d as shutdown script

* README: added drain timing explanation

Author: Bryan Latten

Dockerfile

@@ -1,11 +1,14 @@
 FROM behance/docker-base:2.6-ubuntu-18.04
 MAINTAINER Bryan Latten <[email protected]>
 
+# Use in multi-phase builds, when an init process requests for the container to gracefully exit, so that it may be committed
+# Used with alternative CMD (worker.sh), leverages supervisor to maintain long-running processes
 ENV CONTAINER_ROLE=web \
     CONTAINER_PORT=8080 \
     CONF_NGINX_SITE="/etc/nginx/sites-available/default" \
     CONF_NGINX_SERVER="/etc/nginx/nginx.conf" \
-    NOT_ROOT_USER=www-data
+    NOT_ROOT_USER=www-data \
+    S6_KILL_FINISH_MAXTIME=55000
 
 # Using a non-privileged port to prevent having to use setcap internally
 EXPOSE ${CONTAINER_PORT}

Dockerfile-alpine

@@ -7,7 +7,8 @@ ENV CONTAINER_ROLE=web \
     CONTAINER_PORT=8080 \
     CONF_NGINX_SITE="/etc/nginx/sites-available/default" \
     CONF_NGINX_SERVER="/etc/nginx/nginx.conf" \
-    NOT_ROOT_USER=www-data
+    NOT_ROOT_USER=www-data \
+    S6_KILL_FINISH_MAXTIME=55000
 
 # Using a non-privileged port to prevent having to use setcap internally
 EXPOSE ${CONTAINER_PORT}

Dockerfile-centos

@@ -1,11 +1,14 @@
 FROM behance/docker-base:2.6-centos
 MAINTAINER Bryan Latten <[email protected]>
 
+# Use in multi-phase builds, when an init process requests for the container to gracefully exit, so that it may be committed
+# Used with alternative CMD (worker.sh), leverages supervisor to maintain long-running processes
 ENV CONTAINER_ROLE=web \
     CONTAINER_PORT=8080 \
     CONF_NGINX_SITE="/etc/nginx/sites-available/default" \
     CONF_NGINX_SERVER="/etc/nginx/nginx.conf" \
-    NOT_ROOT_USER=nginx
+    NOT_ROOT_USER=nginx \
+    S6_KILL_FINISH_MAXTIME=55000
 
 # Using a non-privileged port to prevent having to use setcap internally
 EXPOSE ${CONTAINER_PORT}

README.md

@@ -17,59 +17,48 @@ See parent(s) [docker-base](https://github.com/behance/docker-base) for addition
 
 ### Expectations
 
-Applications using this as a container parent must copy their html/app into the `/var/www/html` folder
-NOTE: Nginx is exposed and bound to an unprivileged port, `8080`
+- Applications must copy their html/app into the `/var/www/html` folder
+-   NOTE: Nginx is exposed and bound to an unprivileged port, `8080`
 
 
 ### Security
 
-For Ubuntu-based variants, a convenience script is provided for security-only package updates. To run: `/bin/bash -e /security_updates.sh`
+See parent [configuration](https://github.com/behance/docker-base#security)
 
 
 ### Environment Variables
 
 Variable | Example | Description
 --- | --- | ---
-`SERVER_MAX_BODY_SIZE` | `SERVER_MAX_BODY_SIZE=4M` | Allows the downstream application to specify a non-default `client_max_body_size` configuration for the `server`-level directive in `/etc/nginx/sites-available/default`
-`SERVER_INDEX` | `SERVER_INDEX index.html index.html index.php` | Changes the default pages to hit for folder and web roots
-`SERVER_APP_NAME` | `SERVER_APP_NAME='view'` | Gets appended to the default logging format
-`SERVER_GZIP_OPTIONS` | `SERVER_GZIP_OPTIONS=1` | Allows default set of static content to be served gzipped
-`SERVER_SENDFILE` | `SERVER_SENDFILE=off` | Allows runtime to specify value of nginx's `sendfile` (default, on)
-`SERVER_KEEPALIVE` | `SERVER_KEEPALIVE=30` | Define HTTP 1.1's keepalive timeout
-`SERVER_WORKER_PROCESSES` | `SERVER_WORKER_PROCESSES=4` | Set to the number of cores in the machine, or the number of cores allocated to container
-`SERVER_WORKER_CONNECTIONS` | `SERVER_WORKER_CONNECTIONS=2048` | Sets up the number of connections for worker processes
-`SERVER_CLIENT_HEADER_BUFFER_SIZE` | `SERVER_CLIENT_HEADER_BUFFER_SIZE=16k | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size)
-`SERVER_LARGE_CLIENT_HEADER_BUFFERS` | `SERVER_LARGE_CLIENT_HEADER_BUFFERS=8 16k` | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers)
-`SERVER_CLIENT_BODY_BUFFER_SIZE` | `SERVER_CLIENT_BODY_BUFFER_SIZE=128k` | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size)
-`SERVER_LOG_MINIMAL` | `SERVER_LOG_MINIMAL=1` | Minimize the logging format, appropriate for development environments
-`S6_KILL_FINISH_MAXTIME` | `S6_KILL_FINISH_MAXTIME=1000` | Wait time (in ms) for zombie reaping before sending a kill signal
-`S6_KILL_GRACETIME` | `S6_KILL_GRACETIME=500` | Wait time (in ms) for S6 finish scripts before sending kill signal
+SERVER_MAX_BODY_SIZE | SERVER_MAX_BODY_SIZE=4M | Allows the downstream application to specify a non-default `client_max_body_size` configuration for the `server`-level directive in `/etc/nginx/sites-available/default`
+SERVER_INDEX | SERVER_INDEX index.html index.html index.php | Changes the default pages to hit for folder and web roots
+SERVER_APP_NAME | SERVER_APP_NAME='view' | Gets appended to the default logging format
+SERVER_GZIP_OPTIONS | SERVER_GZIP_OPTIONS=1 | Allows default set of static content to be served gzipped
+SERVER_SENDFILE | SERVER_SENDFILE=off | Allows runtime to specify value of nginx's `sendfile` (default, on)
+SERVER_KEEPALIVE | SERVER_KEEPALIVE=30 | Define HTTP 1.1's keepalive timeout
+SERVER_WORKER_PROCESSES | SERVER_WORKER_PROCESSES=4 | Set to the number of cores in the machine, or the number of cores allocated to container
+SERVER_WORKER_CONNECTIONS | SERVER_WORKER_CONNECTIONS=2048 | Sets up the number of connections for worker processes
+SERVER_CLIENT_HEADER_BUFFER_SIZE | SERVER_CLIENT_HEADER_BUFFER_SIZE=16k | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size)
+SERVER_LARGE_CLIENT_HEADER_BUFFERS | SERVER_LARGE_CLIENT_HEADER_BUFFERS=8 16k | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers)
+SERVER_CLIENT_BODY_BUFFER_SIZE | SERVER_CLIENT_BODY_BUFFER_SIZE=128k | [docs](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size)
+SERVER_LOG_MINIMAL | SERVER_LOG_MINIMAL=1 | Minimize the logging format, appropriate for development environments
+S6_KILL_FINISH_MAXTIME | S6_KILL_FINISH_MAXTIME=55000 | The maximum time (in ms) a script in /etc/cont-finish.d could take before sending a KILL signal to it. Take into account that this parameter will be used per each script execution, it's not a max time for the whole set of scripts. This value has a max of 65535 on Alpine variants.
+S6_KILL_GRACETIME | S6_KILL_GRACETIME=500 | Wait time (in ms) for S6 finish scripts before sending kill signal
 
 
 ### Startup/Runtime Modification
 
-To inject changes just before runtime, shell scripts (ending in .sh) may be placed into the
-`/etc/cont-init.d` folder. For example, the above environment variables are used to drive nginx configuration at runtime.
-As part of the process manager, these scripts are run in advance of the supervised processes. @see https://github.com/just-containers/s6-overlay#executing-initialization-andor-finalization-tasks
+- Environment variables are used to drive nginx configuration at runtime
+- See [here](https://github.com/behance/docker-base#startupruntime-modification) for more advanced options
 
+### Shutdown Behavior
 
-### Advanced Modification
-
-More advanced changes can take effect using the run.d system. Similar to the `/etc/cont-init.d/` script system, any scripts (ending in .sh) in the `/run.d/` folder will be executed ahead of the S6 initialization.
-
-- If run.d script terminates with a non-zero exit code, container will stop, terminating with the script's exit code, unless...
-- If script terminates with exit code of $SIGNAL_BUILD_STOP (99), this will signal the container to stop cleanly. This can be used for multi-stage builds
-
+Graceful shutdown is handled as part of the [existing](https://github.com/behance/docker-base#shutdown-behavior) S6 termination process, using a `/etc/cont-finish.d` script.
+Nginx will attempt to drain active workers, while rejecting new connections. The drain timeout is controlled by `S6_KILL_FINISH_MAXTIME`, which corresponds to the length of time the supervisor will wait for the script to run during shutdown. This value defaults to 55s, which deliberately `less` than an downstream load balancers default max connection length (60s). Each upstream's timeout must be less than the downstream, for sanity and lack of timing precision.
 
 ### Long-running processes (workers + crons)
 
-This container image can be shared between web and non-web processes. An example use case would be
-a web service and codebase that also has a few crons and background workers. To reuse this container for
-those types of workloads:
-
-`docker run {image_id} /worker.sh 3 /bin/binary -parameters -that -binary -receives`
-
-Runs `3` copies of `/bin/binary` that receives the parameters `-parameters -that -binary -receives`
+- See parent [configuration](https://github.com/behance/docker-base#long-running-processes-workers--crons) on reusing container for other purposes.
 
 
 ### Container Organization

container/root/etc/cont-finish.d/00-nginx.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/execlineb -P
+
+foreground { nginx -s quit }
+echo "[finish nginx] shutting down gracefully"

container/root/etc/cont-init.d/10-nginx.sh renamed to container/root/etc/cont-init.d/10-nginx-config.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# Validate that env config overrides are acceptable to nginx
+# Suppress output, unless it fails
+nginx -t &> /dev/null
+
+# Quick and dirty, if the above has failed, re-run it without suppressing output
+if [ $? == 1 ]; then
+  nginx -t
+fi

container/root/etc/cont-init.d/11-nginx-validation.sh

@@ -1,21 +1,17 @@
 #!/usr/bin/execlineb -P
 
-# @see https://github.com/just-containers/s6-overlay/issues/41#issuecomment-99366363
+# TERM broadcast is ignored in order to shut down gracefully,
+# Shutdown implemented in /etc/cont-finish.d
 
-# Nginx uses SIGQUIT to gracefully stop serving traffic.
-trap -x
+trap
 {
   term
   {
-    foreground
-    {
-      nginx -s quit
-    }
-    echo [sigterm-nginx] graceful shutdown complete
+    echo [run nginx] TERM received
   }
   hup
   {
-    echo [signhup-nginx] received, ignored
+    echo [run nginx] HUP received
   }
 }