Fix response body

.travis.yml

@@ -16,8 +16,8 @@ addons:
     - liblmdb-dev
 
 env:
-  - VER_NGINX=1.13.4
-  - VER_NGINX=1.12.1
+  - VER_NGINX=1.13.10
+  - VER_NGINX=1.12.2
 
 before_script:
   - cd ..

AUTHORS

@@ -1 +1,2 @@
 zimmerle = Felipe Zimmerle <[email protected]>
+defanator = Andrei Belov <[email protected]>

CHANGES

@@ -1,5 +1,25 @@
-DD mmm YYYY  - 1.0.0
+v1.0.x - YYYY-MMM-DD (To be released)
+-------------------------------------
+ - Fix processing of response body when gzip compression is enabled
+   [Issue #107 - @turchanov]
+ - Fixed processing of response body chunks in
+   ngx_http_modsecurity_body_filter.
+   [Issue #105 - @turchanov, @defanator]
+ - Fix incorrect handling of request/response body data chain of ngx_buf_t
+   buffers
+   [Issue #104 - @turchanov, @defanator]
+ - Pool pointer is now handled in ngx_http_modsecurity_config_cleanup
+   [Issue #87 - @AirisX, @defanator, @zimmerle]
+ - Fix memory leak in intervention processing
+   [Issue #100 - @defanator]
+ - Emit connector version in error log
+   [Issue #88 - @defanator]
+ - Fixed memory leak on config cleanup.
+   [Issue #80 - @AirisX, @defanator]
+
+
+v1.0.0 - 2017-Dec-20
 --------------------
 
- * First version of the ModSecurity-nginx 
-   [Felipe Zimmerle]
+ - First version of ModSecurity-nginx connector
+

README.md

@@ -5,103 +5,131 @@
 [![](https://raw.githubusercontent.com/ZenHubIO/support/master/zenhub-badge.png)](https://zenhub.com)
 
 
+The ModSecurity-nginx connector is the connection point between nginx and libmodsecurity (ModSecurity v3). Said another way, this project provides a communication channel between nginx and libmodsecurity. This connector is required to use LibModSecurity with nginx. 
 
-The ModSecurity-nginx connector is the connection point between Nginx and libmodsecurity (ModSecurity v3). Said another way, this project provides a communication channel between Nginx and libmodsecurity. This connector is required to use LibModSecurity with Nginx. 
-
-The ModSecurity-nginx connector takes the form of an Nginx module. The module simply serves as a layer of communication between Nginx and ModSecurity.
+The ModSecurity-nginx connector takes the form of an nginx module. The module simply serves as a layer of communication between nginx and ModSecurity.
 
 Notice that this project depends on libmodsecurity rather than ModSecurity (version 2.9 or less).
-libmodsecurity has not reached a stable release candidate, thus, use this project with caution.
 
 ### What is the difference between this project and the old ModSecurity add-on for nginx?
 
 The old version uses ModSecurity standalone, which is a wrapper for
 Apache internals to link ModSecurity to nginx. This current version is closer
 to nginx, consuming the new libmodsecurity which is no longer dependent on
-Apache. As a result, This current version has less dependencies, fewer bugs, and is faster. In addition, Some new functionality is also provided - such as the possibility of use of global rules configuration with per directory/location customizations (e.g. SecRuleRemoveById).
+Apache. As a result, this current version has less dependencies, fewer bugs, and is faster. In addition, some new functionality is also provided - such as the possibility of use of global rules configuration with per directory/location customizations (e.g. SecRuleRemoveById).
 
 
 # Compilation
 
 Before compile this software make sure that you have libmodsecurity installed.
 You can download it from the [ModSecurity git repository](https://github.com/SpiderLabs/ModSecurity). For information pertaining to the compilation and installation of libmodsecurity please consult the documentation provided along with it.
 
-With libmodsecurity installed, you can proceed with the installation of the ModSecurity-nginx connector, which follow the Nginx 3rd party module installation procedure:
+With libmodsecurity installed, you can proceed with the installation of the ModSecurity-nginx connector, which follow the nginx 3rd party module installation procedure:
 
 ```
-./configure --add-module=/path/your/modsecurity-for-nginx
+./configure --add-module=/path/to/ModSecurity-nginx
 ```
 
 Further information about nginx 3rd party add-ons support are available here:
 http://wiki.nginx.org/3rdPartyModules
 
+
 # Usage
 
-ModSecurity for Nginx extends your Nginx configuration directives. It adds four
-new directives and they are:
+ModSecurity for nginx extends your nginx configuration directives.
+It adds four new directives and they are:
 
-modsecurity [On|Off] - This directive turns on or off ModSecurity functionality. Note that
-this configuration directive is no longer related to the SecRule state. Instead, it now serves solely as an Nginx flag to enable or disable the module.
+modsecurity
+-----------
+**syntax:** *modsecurity on | off*
 
-modsecurity_rules_file [<path to rules file>] - This directive indicates the location of the modsecurity configuration file.
+**context:** *http, server, location*
 
-modsecurity_rules_remote [server-key] [<url to rules>] - This directive is used to indicate from where (on the internet) a modsecurity configuration file will be downloaded. It also specifies the key that will be used to authenticate to that server.
+**default:** *off*
 
-modsecurity_rules [<modsecurity rule>] - This directive allows for the direct inclusion of a ModSecurity rule into the Nginx configuration.
+Turns on or off ModSecurity functionality.
+Note that this configuration directive is no longer related to the SecRule state.
+Instead, it now serves solely as an nginx flag to enable or disable the module.
 
+modsecurity_rules_file
+----------------------
+**syntax:** *modsecurity_rules_file &lt;path to rules file&gt;*
 
-### Usage example: injecting rules within nginx configuration
-```
-...
-modsecurity on;
-location / {
-  modsecurity_rules '
-    SecRuleEngine On
-    SecDebugLog /tmp/modsec_debug.log
-    SecDebugLogLevel 9
-    SecRule ARGS "@contains test" "id:1,phase:2,t:trim,block"
-  ';
+**context:** *http, server, location*
+
+**default:** *no*
+
+Specifies the location of the modsecurity configuration file, e.g.:
+
+```nginx
+server {
+    modsecurity on;
+    location / {
+        root /var/www/html;
+        modsecurity_rules_file /etc/my_modsecurity_rules.conf;
+    }
 }
-...
 ```
 
-### Usage example: loading rules from a file and injecting specific configurations per directory/alias
-```
-...
-modsecurity on;
-location / {
-  root /var/www/html;
-  modsecurity_rules_file /etc/my_modsecurity_rules.conf;
-}
-location /ops {
-  root /var/www/html/opts;
-  modsecurity_rules '
-    SecRuleEngine On
-    SecDebugLog /tmp/modsec_debug.log
-    SecDebugLogLevel 9
-    SecRuleRemoveById 10
-  ';
+modsecurity_rules_remote
+------------------------
+**syntax:** *modsecurity_rules_remote &lt;key&gt; &lt;URL to rules&gt;*
+
+**context:** *http, server, location*
+
+**default:** *no*
+
+Specifies from where (on the internet) a modsecurity configuration file will be downloaded.
+It also specifies the key that will be used to authenticate to that server:
+
+```nginx
+server {
+    modsecurity on;
+    location / {
+        root /var/www/html;
+        modsecurity_rules_remote my-server-key https://my-own-server/rules/download;
+    }
 }
-...
 ```
 
-### Usage example: loading rules from a remote server
-```
-...
-modsecurity on;
-location / {
-  root /var/www/html;
-  modsecurity_rules_remote my-server-key https://my-own-server/rules/download;
+modsecurity_rules
+-----------------
+**syntax:** *modsecurity_rules &lt;modsecurity rule&gt;*
+
+**context:** *http, server, location*
+
+**default:** *no*
+
+Allows for the direct inclusion of a ModSecurity rule into the nginx configuration.
+The following example is loading rules from a file and injecting specific configurations per directory/alias:
+
+```nginx
+server {
+    modsecurity on;
+    location / {
+        root /var/www/html;
+        modsecurity_rules_file /etc/my_modsecurity_rules.conf;
+    }
+    location /ops {
+        root /var/www/html/opts;
+        modsecurity_rules '
+          SecRuleEngine On
+          SecDebugLog /tmp/modsec_debug.log
+          SecDebugLogLevel 9
+          SecRuleRemoveById 10
+        ';
+    }
 }
-...
 ```
 
+
 # Contributing
 
 As an open source project we invite (and encourage) anyone from the community to contribute to our project. This may take the form of: new
 functionality, bug fixes, bug reports, beginners user support, and anything else that you
 are willing to help with. Thank you.
 
+
 ## Providing Patches
 
 We prefer to have your patch within the GitHub infrastructure to facilitate our
@@ -129,7 +157,7 @@ You may also take a look at recent bug reports and open issues to get an idea of
 ### Testing your patch
 
 Along with the manual testing, we strongly recommend that you to use the nginx test
-utility to make sure that you patch does not adversely affect the behavior or performance of Nginx. 
+utility to make sure that you patch does not adversely affect the behavior or performance of nginx. 
 
 The nginx tests are available on: http://hg.nginx.org/nginx-tests/ 
 
@@ -142,14 +170,14 @@ $ cd /path/to/nginx/test/repository
 $ TEST_NGINX_BINARY=/path/to/your/nginx prove .
 ```
 
-If you are facing problems getting your added functionality to pass all the  Nginx tests, feel free to contact us or the nginx mailing list at: http://nginx.org/en/support.html
+If you are facing problems getting your added functionality to pass all the nginx tests, feel free to contact us or the nginx mailing list at: http://nginx.org/en/support.html
 
 ### Debugging 
 
 We respect the nginx debugging schema. By using the configuration option
-“--with-debug”  during the nginx configuration you will also  be enabling the
+"--with-debug" during the nginx configuration you will also be enabling the
 connector's debug messages. Core dumps and crashes are expected to be debugged
-in the same fashion that is used to debug Nginx. For further information,
+in the same fashion that is used to debug nginx. For further information,
 please check the nginx debugging information: http://wiki.nginx.org/Debugging
 
 
@@ -169,11 +197,13 @@ version of your libmodsecurity and the version of the nginx connector you are ru
 Please do not publicly report any security issue. Instead, contact us at:
 [email protected] to report the issue. Once the problem is fixed we will provide you with credit for the discovery.
 
+
 ## Feature Request
 
 We would love to discuss any ideas that you may have for a new feature. Please keep in mind this is a community driven project so be sure to contact the community via the mailing list to get feedback first. Alternatively,
 feel free to open GitHub issues requesting for new features. Before opening a new issue, please check if there is an existing feature request for the desired functionality.
 
+
 ## Packing
 
 Having our packages in distros on time is something we highly desire. Let us know if

config

@@ -82,6 +82,30 @@ fi
 
 ngx_addon_name=ngx_http_modsecurity_module
 
+# We must place ngx_http_modsecurity_module after ngx_http_gzip_filter_module
+# in load order list to be able to read response body before it gets compressed
+# (for filter modules later initialization means earlier execution).
+#
+# Nginx implements load ordering only for dynamic modules and only a BEFORE part
+# of "ngx_module_order". So we list all of the modules that come after
+# ngx_http_gzip_filter_module as a BEFORE dependency for
+# ngx_http_modsecurity_module.
+#
+# For static compilation HTTP_FILTER_MODULES will be patched later.
+
+modsecurity_dependency="ngx_http_postpone_filter_module \
+                        ngx_http_ssi_filter_module \
+                        ngx_http_charset_filter_module \
+                        ngx_http_xslt_filter_module \
+                        ngx_http_image_filter_module \
+                        ngx_http_sub_filter_module \
+                        ngx_http_addition_filter_module \
+                        ngx_http_gunzip_filter_module \
+                        ngx_http_userid_filter_module \
+                        ngx_http_headers_filter_module \
+                        ngx_http_copy_filter_module"
+
+
 if test -n "$ngx_module_link"; then
 	ngx_module_type=HTTP_FILTER
 	ngx_module_name="$ngx_addon_name"
@@ -98,7 +122,12 @@ if test -n "$ngx_module_link"; then
         ngx_module_libs="$ngx_feature_libs"
         ngx_module_incs="$ngx_feature_path"
 
-        ngx_module_order="ngx_http_chunked_filter_module ngx_http_v2_filter_module $ngx_module_name ngx_http_range_header_filter_module" 
+        ngx_module_order="ngx_http_chunked_filter_module \
+                          ngx_http_v2_filter_module \
+                          ngx_http_range_header_filter_module \
+                          ngx_http_gzip_filter_module \
+                          $ngx_module_name \
+                          $modsecurity_dependency";
 
 	. auto/module
 else
@@ -128,20 +157,36 @@ fi
 
 #
 # Nginx does not provide reliable way to introduce our module into required
-# place in static ($ngx_module_link=ADDON) compilation mode, so we should
+# place in static ($ngx_module_link=ADDON) compilation mode, so we must
 # explicitly update module "ordering rules".
 #
-# Default runtime location of ngx_http_modsecurity_module is right before
-# ngx_http_chunked_filter_module, but in case if ngx_http_v2_filter_module is
-# compiled in, we should put our module before ngx_http_v2_filter_module in
-# order to support SecRules processing for HTTP/2.0 requests.
-#
 if [ "$ngx_module_link" != DYNAMIC ] ; then
-    pre_module='ngx_http_chunked_filter_module'
-    if [ "$HTTP_V2" = "YES" ]; then
-        pre_module='ngx_http_v2_filter_module'
+    # Reposition modsecurity module to satisfy $modsecurity_dependency
+    # (this mimics dependency resolution made by ngx_add_module() function
+    # though less optimal in terms of computational complexity).
+    modules=
+    found=
+    for module in $HTTP_FILTER_MODULES; do
+        # skip our module name from the original list
+        if [ "$module" = "$ngx_addon_name" ]; then
+            continue
+        fi
+        if [ -z "${found}" ]; then
+            for item in $modsecurity_dependency; do
+                if [ "$module" = "$item" ]; then
+                    modules="${modules} $ngx_addon_name"
+                    found=1
+                    break
+                fi
+            done
+        fi
+        modules="${modules} $module"
+    done
+    if [ -z "${found}" ]; then
+        # This must never happen since ngx_http_copy_filter_module must be in HTTP_FILTER_MODULES
+        # and we stated dependency on it in $modsecurity_dependency
+        echo "$0: error: cannot reposition modsecurity module in HTTP_FILTER_MODULES list"
+        exit 1
     fi
-    HTTP_FILTER_MODULES=`echo $HTTP_FILTER_MODULES | \
-        sed -E "s/$ngx_addon_name/ /g" | \
-        sed -E "s/$pre_module/$pre_module $ngx_addon_name/g"`
+    HTTP_FILTER_MODULES="${modules}"
 fi

release.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+git clean -xfdi
+git submodule foreach --recursive git clean -xfdi
+
+VERSION=`git describe --tags`
+DIR_NAME="modsecurity-nginx-$VERSION"
+TAR_NAME="modsecurity-nginx-$VERSION.tar.gz"
+
+MY_DIR=${PWD##*/}
+
+cd ..
+tar --transform "s/^$MY_DIR/$DIR_NAME/" -cvzf $TAR_NAME --exclude .git $MY_DIR
+
+sha256sum $TAR_NAME > $TAR_NAME.sha256
+gpg --detach-sign -a $TAR_NAME
+
+cd -
+echo $TAR_NAME ": done."
+