Fixes #20889 - Use event mpm module with Apache #753

wbclark · 2022-03-04T22:42:23Z

Some considerations:

There are issues with the older httpd on EL7 that make mpm_event perform worse than it should
This required updates to tuning profiles, and interestingly the prefork values were the same across all tuning sizes. Those apache directives have been updated here with more modern equivalents for mpm_event but these values haven't yet been thoroughly benchmarked

evgeni · 2022-03-05T08:07:32Z

Two questions (which I am not sure myself about):

Should we set EL7 to prefork, or not care as Foreman will be dropping EL7 support soon?
Should we keep the old prefork tunings (my understanding is that defining them in hiera should be a noop while the module isn't loaded), so that when a user decides to revert back to prefork, they get the old settings too?

ekohl

As for the tuning, I think we could even make those the base tuning, if they turn out to be correct.

With the prefork mpm every worker took quite a bit of memory, especially when we had both mod_passenger and mod_wsgi loaded. Nowadays we only have reverse proxying and static file serving so every worker is already very light. For example, using prefork mpm on my machine (no tuning) Apache just takes 15.4 MB of RAM according to systemctl status httpd.

So I did start to read through https://www.liquidweb.com/kb/apache-performance-tuning-mpm-directives/ and it makes me wonder where the right balance is.

wbclark · 2022-03-07T14:57:51Z

Thanks for the suggestions. I've pushed an update which preserves the mod prefork tunings in case that mpm is used, switches to the event mpm only for EL8, and moves the deployment size invariant parameter values into common.yaml (including the existing prefork tunings, which did not vary).

wbclark · 2022-03-07T15:06:11Z

With the prefork mpm every worker took quite a bit of memory, especially when we had both mod_passenger and mod_wsgi loaded. Nowadays we only have reverse proxying and static file serving so every worker is already very light. For example, using prefork mpm on my machine (no tuning) Apache just takes 15.4 MB of RAM according to systemctl status httpd.

Thanks, this makes me think that a good balance is reducing the threadsperchild by a factor of 2 or 4. Vs. what I have now, that would cause a 2-4x increase to the (small) memory footprint of workers while reducing the impact of worker recycling. In this case, the memory consumption should still be significantly smaller compared to the prefork mpm.

ekohl · 2022-03-07T15:23:06Z

We can also consider turning off default mods:
https://github.com/puppetlabs/puppetlabs-apache/blob/e27004b9d7fda5506d28b6e7b8682a04168bdd21/manifests/default_mods.pp#L56-L83

It does mean we need to be careful to include the mods we do use but fewer mods is better for memory usage and security.

wbclark · 2022-03-07T15:26:17Z

So for example:

apache::mod::event::startservers: 16
apache::mod::event::maxrequestworkers: 1024
apache::mod::event::minsparethreads: 128
apache::mod::event::maxsparethreads: 384
apache::mod::event::threadsperchild: 16
apache::mod::event::maxconnectionsperchild: 4096

In this case we'd start up 16 processes, each with 16 threads, so using @ekohl 's numbers this is a cost of around 250 MB of RAM, which increases the memory footprint at idle vs. prefork but allows handling some load spikes without immediately requiring additional workers to be created. Under full load (64 worker processes) the memory consumption should still be about 16x smaller than the prefork mpm.

TODO:
Validate that the above configuration behaves as I expect
Test impact of apache::default_mods: false

ekohl · 2022-03-07T15:39:20Z

I think 250 MB out of the box is quite large. It is a LOT more than we do today. Does startservers need to be that high? I'd assume Apache can scale up servers quite quickly if needed. For reference, my Foreman server today has 4 GB so Apache would take about 6% of the servers memory. I don't think this is really a sane default of our default profile.

wbclark · 2022-03-09T18:23:27Z

I think 250 MB out of the box is quite large. It is a LOT more than we do today. Does startservers need to be that high? I'd assume Apache can scale up servers quite quickly if needed. For reference, my Foreman server today has 4 GB so Apache would take about 6% of the servers memory. I don't think this is really a sane default of our default profile.

It seems odd to me that we didn't previously do anything different with the apache configuration for different tuning sizes, so IMO this is the proper solution.

Work on disabling default modules is ongoing. As expected there are breakages so I'm identifying which are required and re-enabling those. (apache::default_mods can accept not only a boolean type, but also a list of modules, which is a nice feature)

I will check what the memory footprint per worker is like when the minimal set of modules is enabled, and add reasonable sizing based configurations once that figure is obtained.

ehelms · 2022-03-09T20:08:36Z

It seems odd to me that we didn't previously do anything different with the apache configuration for different tuning sizes, so IMO this is the proper solution.

I interpret the previous comments as - we had configuration that worked across all tunings and were a sane default for small on up to large without compromising small tuning for the sake of larger (small/medium being the more common deployments). And that we should not over correct. As that impacts our testing and development environments by potentially requiring more resources than is necessary.

I think you could follow up with different values for the larger tuning profiles as we get performance data to support that. And for the initial pass err on the side of values that target our default tuning profile to match what we had before.

Work on disabling default modules is ongoing. As expected there are breakages so I'm identifying which are required and re-enabling those. (apache::default_mods can accept not only a boolean type, but also a list of modules, which is a nice feature)

Do you have a Redmine for tracking this?

ekohl · 2022-03-10T15:10:19Z

That is indeed what I meant. I'd start conservative with tuning. If Apache can quickly scale up workers and/or threads if needed, I'd start with a low number by default. I'd also check to see how many backend connections we have available. If Puma (for Foreman) and gunicorn (for Pulp) only have a limited number, you only really need to add workers for the additional static files (which I think at the moment are very few, basically only /pub). We don't have 1000 Puma + gunicorn workers available, so 1000 threads is probably too much. It could even open up the path to a Denial of Service attack.

wbclark · 2022-04-28T13:16:03Z

KATELLO 4.4 (apache::default_mods: true and using mod_prefork with the existing tunings):

[root@centos8-stream-katello-4-4 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-04-28 12:19:34 UTC; 44min ago
     Docs: man:httpd.service(8)
 Main PID: 39787 (httpd)
   Status: "Total requests: 16; Idle/Busy workers 100/0;Requests/sec: 0.00601; Bytes served/sec:  55 B/sec"
    Tasks: 9 (limit: 37817)
   Memory: 14.2M
   CGroup: /system.slice/httpd.service
           ├─39787 /usr/sbin/httpd -DFOREGROUND
           ├─39789 /usr/sbin/httpd -DFOREGROUND
           ├─39790 /usr/sbin/httpd -DFOREGROUND
           ├─39791 /usr/sbin/httpd -DFOREGROUND
           ├─39792 /usr/sbin/httpd -DFOREGROUND
           ├─39793 /usr/sbin/httpd -DFOREGROUND
           ├─39794 /usr/sbin/httpd -DFOREGROUND
           ├─39795 /usr/sbin/httpd -DFOREGROUND
           └─39796 /usr/sbin/httpd -DFOREGROUND

Apr 28 12:19:34 centos8-stream-katello-4-4.mercury.example.com systemd[1]: Starting The Apache HTTP Server...
Apr 28 12:19:34 centos8-stream-katello-4-4.mercury.example.com systemd[1]: Started The Apache HTTP Server.
Apr 28 12:19:34 centos8-stream-katello-4-4.mercury.example.com httpd[39787]: Server configured, listening on: port 80, port 443

[root@centos8-stream-katello-4-4 ~]# ps aux | grep -v grep | grep httpd
root       39787  0.0  0.1 183628 10996 ?        Ss   12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39789  0.0  0.1 200504 11220 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39790  0.0  0.1 200504 11404 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39791  0.0  0.1 200700 11540 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39792  0.0  0.1 200700 11580 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39793  0.0  0.1 200700 11556 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39794  0.0  0.1 200700 11540 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39795  0.0  0.1 200508 11280 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND
apache     39796  0.0  0.1 200520 11380 ?        S    12:19   0:00 /usr/sbin/httpd -DFOREGROUND

[root@centos8-stream-katello-4-4 ~]# for pid in $(ps aux | grep -v grep | grep httpd | awk '{print $2}'); do ps -T -p $pid ; done
    PID    SPID TTY          TIME CMD
  39787   39787 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39789   39789 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39790   39790 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39791   39791 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39792   39792 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39793   39793 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39794   39794 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39795   39795 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  39796   39796 ?        00:00:00 httpd

(Key metrics: 9 processes, 9 threads, 14.2M Memory)

KATELLO NIGHTLY (including prior apache::default_mods: false change, apache::mpm_module: 'event', and the tunings present here):

[root@centos8-stream-katello-nightly ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-04-28 11:07:25 UTC; 2h 3min ago
     Docs: man:httpd.service(8)
 Main PID: 40164 (httpd)
   Status: "Total requests: 16; Idle/Busy workers 100/0;Requests/sec: 0.00215; Bytes served/sec:   2 B/sec"
    Tasks: 37 (limit: 37817)
   Memory: 13.2M
   CGroup: /system.slice/httpd.service
           ├─40164 /usr/sbin/httpd -DFOREGROUND
           ├─40166 /usr/sbin/httpd -DFOREGROUND
           └─40167 /usr/sbin/httpd -DFOREGROUND

Apr 28 11:07:25 centos8-stream-katello-nightly.mercury.example.com systemd[1]: Starting The Apache HTTP Server...
Apr 28 11:07:25 centos8-stream-katello-nightly.mercury.example.com systemd[1]: Started The Apache HTTP Server.
Apr 28 11:07:25 centos8-stream-katello-nightly.mercury.example.com httpd[40164]: Server configured, listening on: port 80, port 443

[root@centos8-stream-katello-nightly ~]# ps aux | grep -v grep | grep httpd
root       40164  0.0  0.1 120200 10756 ?        Ss   11:07   0:00 /usr/sbin/httpd -DFOREGROUND
apache     40166  0.0  0.1 1267240 11636 ?       Sl   11:07   0:00 /usr/sbin/httpd -DFOREGROUND
apache     40167  0.0  0.1 1267368 10796 ?       Sl   11:07   0:00 /usr/sbin/httpd -DFOREGROUND

[root@centos8-stream-katello-nightly ~]# for pid in $(ps aux | grep -v grep | grep httpd | awk '{print $2}'); do ps -T -p $pid ; done
    PID    SPID TTY          TIME CMD
  40164   40164 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  40166   40166 ?        00:00:00 httpd
  40166   40187 ?        00:00:00 httpd
  40166   40188 ?        00:00:00 httpd
  40166   40189 ?        00:00:00 httpd
  40166   40190 ?        00:00:00 httpd
  40166   40191 ?        00:00:00 httpd
  40166   40192 ?        00:00:00 httpd
  40166   40193 ?        00:00:00 httpd
  40166   40194 ?        00:00:00 httpd
  40166   40195 ?        00:00:00 httpd
  40166   40196 ?        00:00:00 httpd
  40166   40197 ?        00:00:00 httpd
  40166   40198 ?        00:00:00 httpd
  40166   40199 ?        00:00:00 httpd
  40166   40200 ?        00:00:00 httpd
  40166   40201 ?        00:00:00 httpd
  40166   40202 ?        00:00:00 httpd
  40166   40203 ?        00:00:00 httpd
    PID    SPID TTY          TIME CMD
  40167   40167 ?        00:00:00 httpd
  40167   40170 ?        00:00:00 httpd
  40167   40171 ?        00:00:00 httpd
  40167   40172 ?        00:00:00 httpd
  40167   40173 ?        00:00:00 httpd
  40167   40174 ?        00:00:00 httpd
  40167   40175 ?        00:00:00 httpd
  40167   40176 ?        00:00:00 httpd
  40167   40177 ?        00:00:00 httpd
  40167   40178 ?        00:00:00 httpd
  40167   40179 ?        00:00:00 httpd
  40167   40180 ?        00:00:00 httpd
  40167   40181 ?        00:00:00 httpd
  40167   40182 ?        00:00:00 httpd
  40167   40183 ?        00:00:00 httpd
  40167   40184 ?        00:00:00 httpd
  40167   40185 ?        00:00:00 httpd
  40167   40186 ?        00:00:00 httpd

(Key metrics: 3 processes, 37 threads, 13.2M Memory)

wbclark · 2022-04-28T13:33:26Z

@ehelms @ekohl @evgeni

I've updated this with the intention of more closely matching the previous behavior. The thread count is still higher than before, but within the same order of magnitude and as you can see the memory usage is actually lower at idle.

The reason for the higher thread count is that mpm_event uses worker recycling to prevent memory from growing without bound; rather than get into a situation where maxrequestsperchild is reached and processing of new requests must be temporarily halted while waiting on the worker to recycle, I've configured minsparethreads so that there should always be an extra worker ready to process new requests seamlessly while the previous worker recycles.

Note that threadsperchild has a default value of 25 and that is reduced here to 16. So with the default, even reducing to a single worker here would exceed the previous configuration. The tradeoff with increasing threadsperchild is that it achieves lower memory usage per thread due to shared memory, the tradeoff being that recycling becomes more expensive.

ehelms · 2022-04-28T15:00:52Z

config/foreman.hiera/tuning/sizes/medium.yaml

@@ -1,4 +1,8 @@
 ---
+apache::mod::event::serverlimit: 64
+apache::mod::event::maxrequestworkers: 1024
+apache::mod::event::maxrequestsperchild: 4000


This seems quite a jump from default to medium, what is your thinking there?

If the question is about ServerLimit and MaxRequestWorkers, this tuning profile already has apache::mod::prefork::serverlimit: 1024 which allows up to 1024 single threaded httpd worker processes.

For event_mpm case, ServerLimit * ThreadsPerChild = MaxRequestWorkers and we have ThreadsPerChild = 16, so matching the # of threads we have already for mpm_prefork implies ServerLimit * 16 = 1024, so ServerLimit =64

If the question is about MaxRequestsPerChild, the default is 0 which disables worker recycling entirely. We previously added apache::mod::prefork::maxrequestsperchild: 4000 to protect against memory leaks, so matching this value with the event mpm makes sense because threads are able to share memory within each worker process and this is a per worker setting.

I think it's a likely oversight that MaxRequestsPerChild was previously not set in the default profile -- if we are concerned about the possibility of memory leaks then we should protect against them in the default profile as well. If you agree, I'll push an update adding that to the default as well.

I was also surprised to find that the same limits are already used in medium through extra-extra-large deployment sizes, and was tempted to actually scale them up and down with the profile sizes, but opted instead to heed the previous advice about changing too much at one time.

Appreciate the detailed explanation, that all makes sense. I wouldn't worry about updating prefork configuration - we made it this far and EL7 support is set to be removed really soon.

FWIW, now that we don't run mod_passenger and mod_wsgi inside Apache, I'm less concerned about memory leaks and we could see whether increasing MaxRequestsPerChild or even setting it to 0 would make sense.

But obviously, this doesn't belong into THIS PR ;-)

ehelms

@evgeni you previously commented - mind having a look?

evgeni · 2022-04-29T09:11:12Z

config/foreman.hiera/family/RedHat-8.yaml

@@ -1,4 +1,8 @@
 ---
+# EL8 only due to issues with older Apache event mpm on EL7. When EL7
+# support is dropped, this should move to tuning/common.yaml
+apache::mpm_module: 'event'


Would setting apache::mpm_module: 'event' in config/foreman.hiera/common.yaml (or tuning/common.yaml, idk what's better) and then apache::mpm_module: 'prefork' in config/foreman.hiera/family/RedHat-7.yaml get us the same result?

The benefit would be that this then also would apply to Debian/Ubuntu installations.

evgeni

I'm cool with postponing the Debian part to a later PR.

ehelms · 2022-04-29T09:20:30Z

I like this idea as it also sets the future base state we want and makes EL7 the odd duck which will be dropped soon.

…

On Fri, Apr 29, 2022, 5:11 AM Evgeni Golov ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In config/foreman.hiera/family/RedHat-8.yaml <#753 (comment)> : > @@ -1,4 +1,8 @@ --- +# EL8 only due to issues with older Apache event mpm on EL7. When EL7 +# support is dropped, this should move to tuning/common.yaml +apache::mpm_module: 'event' Would setting apache::mpm_module: 'event' in config/foreman.hiera/common.yaml (or tuning/common.yaml, idk what's better) and then apache::mpm_module: 'prefork' in config/foreman.hiera/family/RedHat-7.yaml get us the same result? The benefit would be that this then also would apply to Debian/Ubuntu installations. — Reply to this email directly, view it on GitHub <#753 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHT47OYSB7MOOVCCRYCZTVHORTZANCNFSM5P6XTQXQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

config/foreman.hiera/family/RedHat-7.yaml

wbclark · 2022-04-29T11:30:45Z

Great idea. I've added apache::mpm_module: 'event' to foreman.hiera/common.yaml to serve as the default. This is overridden to apache::mpm_module: 'prefork' for EL7 in foreman.hiera/family/RedHat-7.yaml

evgeni · 2022-04-29T11:44:09Z

ci failure is unrelated, blame @evgeni! theforeman/puppet-foreman_proxy#748

evgeni · 2022-04-29T12:02:12Z

[test foreman-installer]

evgeni

Thanks a ton @wbclark!

theforeman-bot added Needs testing Not yet reviewed labels Mar 4, 2022

ekohl reviewed Mar 5, 2022

View reviewed changes

wbclark force-pushed the 20889_event_mpm branch from 5ba537c to 3a79b5c Compare March 7, 2022 14:55

wbclark force-pushed the 20889_event_mpm branch from 3a79b5c to 4852655 Compare March 10, 2022 15:03

wbclark mentioned this pull request Mar 10, 2022

Fixes #34590 - Enable only minimum apache modules #754

Merged

wbclark force-pushed the 20889_event_mpm branch from 4852655 to 0c7c592 Compare March 10, 2022 15:11

wbclark force-pushed the 20889_event_mpm branch from 0c7c592 to 21da6ef Compare April 28, 2022 10:00

ehelms reviewed Apr 28, 2022

View reviewed changes

ehelms approved these changes Apr 29, 2022

View reviewed changes

theforeman-bot removed the Not yet reviewed label Apr 29, 2022

evgeni reviewed Apr 29, 2022

View reviewed changes

evgeni approved these changes Apr 29, 2022

View reviewed changes

wbclark force-pushed the 20889_event_mpm branch from 21da6ef to bf4ef4f Compare April 29, 2022 11:26

evgeni reviewed Apr 29, 2022

View reviewed changes

config/foreman.hiera/family/RedHat-7.yaml Show resolved Hide resolved

wbclark force-pushed the 20889_event_mpm branch from bf4ef4f to d837f45 Compare April 29, 2022 11:48

Fixes #20889 - Use event mpm module with Apache

1ca9b3c

wbclark force-pushed the 20889_event_mpm branch from d837f45 to 1ca9b3c Compare April 29, 2022 11:54

evgeni approved these changes Apr 29, 2022

View reviewed changes

evgeni merged commit 640dfc6 into theforeman:develop Apr 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #20889 - Use event mpm module with Apache #753

Fixes #20889 - Use event mpm module with Apache #753

wbclark commented Mar 4, 2022

evgeni commented Mar 5, 2022

ekohl left a comment

wbclark commented Mar 7, 2022

wbclark commented Mar 7, 2022 •

edited

Loading

ekohl commented Mar 7, 2022

wbclark commented Mar 7, 2022

ekohl commented Mar 7, 2022

wbclark commented Mar 9, 2022

ehelms commented Mar 9, 2022

ekohl commented Mar 10, 2022

wbclark commented Apr 28, 2022 •

edited

Loading

wbclark commented Apr 28, 2022 •

edited

Loading

ehelms Apr 28, 2022

wbclark Apr 28, 2022

ehelms Apr 29, 2022

evgeni Apr 29, 2022

ehelms left a comment

evgeni Apr 29, 2022

evgeni left a comment

ehelms commented Apr 29, 2022 via email

wbclark commented Apr 29, 2022

evgeni commented Apr 29, 2022

evgeni commented Apr 29, 2022

evgeni left a comment

Fixes #20889 - Use event mpm module with Apache #753

Fixes #20889 - Use event mpm module with Apache #753

Conversation

wbclark commented Mar 4, 2022

evgeni commented Mar 5, 2022

ekohl left a comment

Choose a reason for hiding this comment

wbclark commented Mar 7, 2022

wbclark commented Mar 7, 2022 • edited Loading

ekohl commented Mar 7, 2022

wbclark commented Mar 7, 2022

ekohl commented Mar 7, 2022

wbclark commented Mar 9, 2022

ehelms commented Mar 9, 2022

ekohl commented Mar 10, 2022

wbclark commented Apr 28, 2022 • edited Loading

wbclark commented Apr 28, 2022 • edited Loading

ehelms Apr 28, 2022

Choose a reason for hiding this comment

wbclark Apr 28, 2022

Choose a reason for hiding this comment

ehelms Apr 29, 2022

Choose a reason for hiding this comment

evgeni Apr 29, 2022

Choose a reason for hiding this comment

ehelms left a comment

Choose a reason for hiding this comment

evgeni Apr 29, 2022

Choose a reason for hiding this comment

evgeni left a comment

Choose a reason for hiding this comment

ehelms commented Apr 29, 2022 via email

wbclark commented Apr 29, 2022

evgeni commented Apr 29, 2022

evgeni commented Apr 29, 2022

evgeni left a comment

Choose a reason for hiding this comment

wbclark commented Mar 7, 2022 •

edited

Loading

wbclark commented Apr 28, 2022 •

edited

Loading

wbclark commented Apr 28, 2022 •

edited

Loading