SecRuleEngine off - still performance numbers drop

[mitha]

Hi

modsecurity version - 2.7.5
nginx versoin - 1.4.2
owasp rules set - 2.2.8

I am using modsecurity as WAF inline with nginx serving as reverse proxy.
I am trying to measure the performance of nginx with mod security with SecRuleEngine - off,detection,on and without modsecurity.
What i find is as soon as modsecurity is introduced in nginx.conf ,though SecRuleEngine is set to off,the performance - transaction per second - TPS drops.There is no significant difference between SecRuleEngine being set to on/off/detection in TPS measurement.
I could see in the modsecurity debug log that the transactions are not processed

My question is :

What i dont understand is,even though the transactions are not processed in modsecurity when SecRuleEngine is off why the performance should drop when it is introduced in nginx?

configuration TPS
nginx only 3594
nginx+modsec+SecRuleEngine-off 1399
nginx+modsec+SecRuleEngine-detection 342
nginx+modsec+SecRuleEngine-on 580

Load generator/Stress tool used - weighttp(WG)
Configuration:
The following configuration is kept on for all the below given tests
total request - 500000
connection rate - 100/s
keep alive - on
file size fetched - 1kb
cpu - 1

All the fetches are genuine ,No threats were injected
VM size - small
nginx running process - 1 each master,worker,cache manager process

[zimmerle]

Depends on: #645

[mitha]

Had incorporated your fix - still the performance doesn't seem to increase.
What i find is when the modsec is disbaled/off mode the context created is definitely an overhead as anyways the modsec is disabled and we are not going to process the request.
So why not check the config before setting up the context- whether the modsec is in enabled mode then create context based on the mode and process ,else simply return without setting up any transaction/request context ?

[mitha]

I measure performance by setting RuleEngine to either of 3 options Off ,Detection,On .
Irrespective of the RuleEngine option the performance is low say the TransactionPerSecond.
Ideally when RuleEngine is set to Off, the performance should almost match up when modsecurity config itself is off in nginx config
as the requests needn't be parsed by modsecurity module.
But what i perceive is irrespective of the RuleEngine option, the requests are parsed which takes a toll on the performance
I had done a small fix to mitigate this and added the same for reference and also tried running performance test could see a significant performance improvement.
Please validate the fix .

*************** ngx_http_modsecurity_handler(ngx_http_re
*** 972,977 ****
--- 974,986 ----

  cf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity);

• if (cf != NULL) {
  

•     if(cf->config->is_enabled == 0) {
  

•         return NGX_DECLINED;
  

•     }
  

• }
  

  • /* Process only main request */
    if (r != r->main || !cf->enable) {
    return NGX_DECLINED;

[mitha]

Performance tool used tool is Apache Bench

Configuation TPS measured in iteration 1 2 3 4 5

nginx+modsec - off 1379.49 1397.04 1501.31 1113.12 1304.99
nginx+modsec - on RuleEngine - off 1399.19 1377.88 1297.99 1297.3 1459.25
nginx+modsec - on RuleEngine - on 283.1 290.12 308.19 300.71 300.55
nginx+modsec - on RuleEngine - on 303.45 301.89 303.79 300.31 286.62

[zimmerle]

Hi @mitha,

Nice that you are interested on ModSec' performance, specially on Nginx which is using our standalone module. We will love to have something incorporate to the project to measure those results in a systematic fashion, using scripts or so. That way we will be able to make comparisons between different versions in a easy way. If you are interested in provide such scripts or ideas, lets open an issue to discuss that.

I understand that in the scenario that you have described the results seems a little unhoped. But, there is an explanation regarding the fact that "SecRuleEngine Off" is messing with the performance. We have two different methods to "disable" Nginx' ModSecurity module, the first is using the ModSecurityEnabled and the second is by using the SecRuleEngine. I expect that while setting ModSecurityEnabled to Off the results will be very similar to the results without ModSecurity. However, using the SecRuleEngine, the results will be like you've showed.

Let me explain the difference, ModSecurityEnabled is saved under the structure:

typedef struct {
    ngx_flag_t                  enable;
    directory_config            *config;

    ngx_str_t                   *file;
    ngx_uint_t                   line;
} ngx_http_modsecurity_loc_conf_t;

This ngx_flag_t enable is used by many others Nginx modules, such as:

• Gzip (File: src/http/modules/ngx_http_gzip_filter_module.c line 16)
• Auto index (src/http/modules/ngx_http_autoindex_module.c line 40)

Like those modules, whenever enabled is set to Off, it is expected that ModSecurity does not significant impact on the performance. The SecRuleEngine is a little different, it is rule in first place which demands ModSecurity to actually process it. Whenever SecRuleEngine is set to Off, it means that rules processing should not be working, it does not complete disable ModSecurity. Remember that SecRuleEngine can have a third value different from On and Off. DetectionOnly places the rules in verbose mode but not execute any disruptive action. While using SecRuleEngine to DetectionOnly it is also expected a performance harm.

Looking for the labels that you have placed in your measurements results, i am wondering if the correct comparison will be something like (at least looking at the code like it is today):

• nginx+modsec (enabled) - SecRuleEngine off
• nginx+modsec (enabled) - SecRuleEngine on
• nginx+modsec (enabled) - SecRuleEngine DetectionOnly
• nginx+modsec (disabled)

This open space for other discussion, which is related to your patch: what the practical different between SecRuleEngine off and ModSecurityEnabled Off, Can/Should we have only one place to set it to On/Off or DetectionOnly like we have in Apache? What is your opinion?

Looking at your patch it almost placed the SecRuleEngine in equal foot to ModSecurityEnabled. If it is the case, we can remove one of those. What do you think?

Thanks,
F.

[mitha]

Hi Zimmerle

Good to get an almost immediate reply
First and foremost thing ..thanks for the same

Now on to the discussion topic -

To measure performance:
I had used tools ApacheBench and weighttp
Ref: http://gwan.com/en_apachebench_httperf.html
The link is awesome and gives the information of how to use the tools
So this is my take on to test the performance
1 . Fetch a 1KB file from the webserver
2 . Fetch it with 100 connections concurrently and
3. Huge number of requests say 500000
4. with keep alive request

This is how i used it
weighttp -n 500000 -c 100 -t 1 -k "http://ec2-54-235-2-8.compute-1.amazonaws.com/test_dir/1kb.txt"

ab -n 500000 -c 100 -k "http://ec2-54-235-2-8.compute-1.amazonaws.com/test_dir/1kb.txt"

TPS is the important factor that we take into account as a result of performance measurement and that is the deciding factor for choosing various configuration

You are right about the comparison and i also did the same
There is a typo instead of "Detection" i had put as "On" itself.This is what i measured actually
nginx+modsec - off
nginx+modsec - on RuleEngine - off
nginx+modsec - on RuleEngine - on
nginx+modsec - on RuleEngine - Detection

And regarding having SecRuleEngine and also ModSecurityEnabled , i am not sure what other significant purpose SecRuleEngine have other than having either of the 3 options.
From your explanation i see that setting SecRuleEngine to off does processes the request and also checked in the log,but doesn't really do anything which actually degrades the performance.
Its rather kind of confusing to have two options say SecRuleEngine and ModSecurityEnabled and the combination gives different performance numbers.
As you suggested it is better to have one field and have this 3 options on/off/detection similar to Apache so that new users wouldn't be misled..
As the nginx config have ModSecurityEnabled it would be evident in the firsthand to have that field with the 3 options on/off/detection and removing SecRuleEngine.
Share your opinion on the same taking into account the compatibility of modsecurity with other servers as well.

Thanks and Regards
mitha

[mitha]

Hi Zimmerle

Good to get an almost immediate reply
First and foremost thing ..thanks for the same

Now on to the discussion topic -

To measure performance:
I had used tools ApacheBench and weighttp
Ref: http://gwan.com/en_apachebench_httperf.html
The link is awesome and gives the information of how to use the tools
So this is my take on to test the performance
1 . Fetch a 1KB file from the webserver
2 . Fetch it with 100 connections concurrently and
3. Huge number of requests say 500000
4. with keep alive request

This is how i used it
weighttp -n 500000 -c 100 -t 1 -k "
http://ec2-54-235-2-8.compute-1.amazonaws.com/test_dir/1kb.txt"

ab -n 500000 -c 100 -k "
http://ec2-54-235-2-8.compute-1.amazonaws.com/test_dir/1kb.txt"

TPS is the important factor that we take into account as a result of
performance measurement and that is the deciding factor for choosing
various configuration

You are right about the comparison and i also did the same
There is a typo instead of "Detection" i had put as "On" itself.This is
what i measured actually
nginx+modsec - off
nginx+modsec - on RuleEngine - off
nginx+modsec - on RuleEngine - on
nginx+modsec - on RuleEngine - Detection

And regarding having SecRuleEngine and also ModSecurityEnabled , i am not
sure what other significant purpose SecRuleEngine have other than having
either of the 3 options.
From your explanation i see that setting SecRuleEngine to off does
processes the request and also checked in the log,but doesn't really do
anything which actually degrades the performance.
Its rather kind of confusing to have two options say SecRuleEngine and
ModSecurityEnabled and the combination gives different performance numbers.
As you suggested it is better to have one field and have this 3 options
on/off/detection similar to Apache so that new users wouldn't be misled..
As the nginx config have ModSecurityEnabled it would be evident in the
firsthand to have that field with the 3 options on/off/detection and
removing SecRuleEngine.
Share your opinion on the same taking into account the compatibility of
modsecurity with other servers as well.

Thanks and Regards
mitha

On Fri, Jan 24, 2014 at 8:19 AM, Felipe Zimmerle
[email protected]:

Hi @mitha https://github.com/mitha,

Nice that you are interested on ModSec' performance, specially on Nginx
which is using our standalone module. We will love to have something
incorporate to the project to measure those results in a systematic
fashion, using scripts or so. That way we will be able to make comparisons
between different versions in a easy way. If you are interested in provide
such scripts or ideas, lets open an issue to discuss that.

I understand that in the scenario that you have described the results
seems a little unhoped. But, there is an explanation regarding the fact
that "SecRuleEngine Off" is messing with the performance. We have two
different methods to "disable" Nginx' ModSecurity module, the first is
using the ModSecurityEnabled and the second is by using the
SecRuleEngine. I expect that while setting ModSecurityEnabled to _Off_the results will be very similar to the results without ModSecurity.
However, using the SecRuleEngine, the results will be like you've showed.

Let me explain the difference, ModSecurityEnabled is saved under the
structure:

typedef struct {
ngx_flag_t enable;
directory_config *config;

ngx_str_t                   *file;
ngx_uint_t                   line;

} ngx_http_modsecurity_loc_conf_t;

This ngx_flag_t enable is used by many others Nginx modules, such as:

• Gzip (File: src/http/modules/ngx_http_gzip_filter_module.c line 16)
• Auto index (src/http/modules/ngx_http_autoindex_module.c line 40)

Like those modules, whenever enabled is set to Off, it is expected that
ModSecurity does not significant impact on the performance. The
SecRuleEngine is a little different, it is rule in first place which
demands ModSecurity to actually process it. Whenever SecRuleEngine is set
to Off, it means that rules processing should not be working, it does not
complete disable ModSecurity. Remember that SecRuleEngine can have a third
value different from On and Off. DetectionOnly places the rules in verbose
mode but not execute any disruptive action. While using SecRuleEngine to
DetectionOnly it is also expected a performance harm.

Looking for the labels that you have placed in your measurements results,
i am wondering if the correct comparison will be something like (at least
looking at the code like it is today):

• nginx+modsec (enabled) - SecRuleEngine off
• nginx+modsec (enabled) - SecRuleEngine on
• nginx+modsec (enabled) - SecRuleEngine DetectionOnly
• nginx+modsec (disabled)

This open space for other discussion, which is related to your patch: what
the practical different between SecRuleEngine off and ModSecurityEnabled
Off, Can/Should we have only one place to set it to On/Off or
DetectionOnly like we have in Apache? What is your opinion?

Looking at your patch it almost placed the SecRuleEngine in equal foot to
ModSecurityEnabled. If it is the case, we can remove one of those. What do
you think?

Thanks,
F.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/644#issuecomment-33193112
.

[wemz]

The same problem.running a few times, modsecurity high memory usage 97.6%
with nginx+modsec - on RuleEngine - off

1500~~2000 users
Have any idea?

[zimmerle]

Marking as won't fix in 2.x. That is already Ok in libmodsecurity.

https://github.com/SpiderLabs/ModSecurity/tree/v3/master