nginx_refacorting leaks memory after reloading nginx

[LanShi]

Hi,

I test Nginx1.8.0 with nginx_refactoring branch module .and find memory leak still exists after reloading nginx.
This is my configurations:

nginx.conf:

http {

server {
    listen       80;
    server_name  localhost;

    location / {
      ModSecurityEnabled on;
      ModSecurityConfig modsecurity.conf;
      proxy_pass http://localhost:8080;
    }
}

}

modsecurity.conf:
SecDebugLog /var/log/modsec_debug.log
SecDebugLogLevel 9
SecRuleEngine On
include /usr/local/nginx/conf/modsecurity-crs/modsecurity_crs_10_setup.conf
include /usr/local/nginx/conf/modsecurity-crs/base_rules/modsecurity_crs_41_sql_injection_attacks.conf

I will be glad to provide other information if necessary.
Thanks

[zimmerle]

Hi @LanShi,

Thanks for the report.

This issue is known. Patch proposed at #815, but not finished yet.

Also leaking file descriptors, as described here: #838

[daniilyar-confyrm]

We face the said memory leak with ModSec 2.9.0 compiled from code with Nginx 1.8.0. We are running Nginx+ModSec in pair with Consul-template, which reloads Nginx config anytime when backend service config changes in Consul. Our Nginx servers are running out or memory after 100-200 config reloads, which totally breaks the stability of our (pretty dynamic) infra.

So I am wondering why, after one year, the fix is still not present in a stable branch.. What blocks the release of this fix?

I will be glad to help, e.g. I can help with reproducing this issue by providing a Docker image with erroneous Nginx + Modsec with reproducer script.
I am not a C++ pro, but I can try to help with patching existing stable codebase with this patch and testing it in a real env..

[zimmerle]

Hi @daniilyar-confyrm,

I answered your comments in the pull request. Btw, did you had the chance to test the pull request?

[daniilyar-confyrm]

@zimmerle, I tested the 'nginx_refactoring' branch under the load at the real env.
It was working well in our dynamic infra and satisfied us from the performance perspective (I tested with default OWASP rule set).

But using this branch at PROD is not acceptable for us because except for the said fix there is a lot of other changes not qualified by community (who knows how many security issues these refactoring changes add). So I forked the 2.9.0 version and applied only critical fixes from 'nginx_refactoring' branch on the top of it: https://github.com/daniilyar/ModSecurity.

This fork is tested under the load at our dynamic env and satisfies our needs. We are going to merge all farther critical fixes to it until new 'clear and shiny' ModSec 3.0 (http://blog.zimmerle.org/2016/01/an-overview-of-upcoming-libmodsecurity.html) will be at least near-to-PROD-ready.

[ebaldonado]

Hi @daniilyar-confyrm

I have tested the https://github.com/daniilyar/ModSecurity and use nginx 1.9.12 and its work great on reload, however i have tried the restart of nginx and it seem some leak happened. the Semaphore Arrays was not release when restart was done. would you be able to confirm on your build if such issue does not exist. As i might just have screw up the complilation

--prefix=/etc/nginx --sbin-path=/local/opt/nginx/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-http_geoip_module --add-module=../ModSecurity-master/nginx/modsecurity --with-debug

restart_log.txt

[daniilyar-confyrm]

@heatburn, thank you, I will take a look in nearest days

[ebaldonado]

@daniilyar-confyrm

I have run a couple of test and it appears that the issue happened when the configtest is run. The init script i am using to restart include a configtest -q before stop and start. Manually running nginx -t yield same issue.

[zimmerle]

Won't fix in 2.x and fixed in libmodsecurity

Further information available here - https://github.com/SpiderLabs/ModSecurity-nginx