Feature: set limits for exec driver

[ashald]

Is it possible to configure Linux limits (as in https://linux.die.net/man/5/limits.conf) with Nomad when exec driver is used? It'd be great if it was possible to set them on the task level or at least on the client config level.

[dadgar]

What exactly are you trying to accomplish with that? We generally try to avoid leaking to many low level details into the job spec!

[ashald]

Some processes need more resources (that controlled via limits) than others so I thought that it might be a nice fit for driver-specific options within job description. OTOH one may argue that it can be set on user level and different users can be used instead.

[dadgar]

Which limits are you interested, process and fd limits?

[ashald]

Yeah, nofile and nproc.

[ashald]

Just for the record, Docker has an option to tweak limits:

--ulimit ulimit                         Ulimit options (default [])

If Nomad's exec driver aim to be on par with Docker (at least that is my impression) then it might make sense to implement this feature.

[rokka-n]

That would be really nice to have since one misbehaving container can easily bring down a node by leaking connections.

[tantra35]

Any chance that this will be implemented?

[dadgar]

@tantra35 Yes it will be. Not currently slated for a particular release.

[prologic]

We are currently blocked by this in our production setup and cannot run ElasticSearch using the exec driver as ES apparently requires a ulimit of at least 65535.

[prologic]

OTOH ES's requirement of 65k open fds seems absurd to me :/ (separate issue)

[prologic]

Work-around (if running Nomad via systemd based systems):

Set LimitNOFILE=65536 in your [Service] section of your Nomad systemd unit.

[prologic]

CGroups (which the exec driver uses through libcontainers/runc) inherits the RLimits of the parent process (in this case nomad).

[mr-karan]

I tried a bunch of workarounds to apply sysctl limits inside an exec task, but none seem to be applied correctly:

Inside the task. /etc/sysctl.conf (which is present inside exec task, chroot_env is the default)

net.core.somaxconn=60000

However, it's not "in use":

cat /proc/sys/net/core/somaxconn
4096

Running sysctl -p is also not helping:

sysctl -p
sysctl: setting key "fs.file-max", ignoring: Read-only file system
sysctl: setting key "net.ipv4.ip_local_port_range", ignoring: Read-only file system
sysctl: setting key "net.netfilter.nf_conntrack_max": Read-only file system
sysctl: setting key "net.ipv4.tcp_syncookies", ignoring: Read-only file system
sysctl: setting key "net.core.somaxconn", ignoring: Read-only file system
sysctl: setting key "net.ipv4.tcp_max_syn_backlog", ignoring: Read-only file system
sysctl: setting key "net.ipv4.tcp_slow_start_after_idle", ignoring: Read-only file system
sysctl: setting key "net.ipv4.tcp_fin_timeout", ignoring: Read-only file system
sysctl: setting key "net.ipv4.tcp_keepalive_time", ignoring: Read-only file system
sysctl: setting key "vm.overcommit_memory", ignoring: Read-only file system

---

However, I was able to set a higher limit for max open files by the process, by adjusting LimitNOFILE in the systemd service.

sed -i 's/LimitNOFILE=65536/LimitNOFILE=900000/g' /lib/systemd/system/nomad.service

So, now ulimit correctly shows the max number of open files for that process:

ulimit -n
900000

But other sysctl limits have no effect until I restart Nomad agent when an alloc is already running here:

cat /proc/sys/net/core/somaxconn
4096

On restarting the agent:

sudo systemctl restart nomad

cat /proc/sys/net/core/somaxconn
60000

This seems like a weird edge case where Nomad applies the limits inside the task only after a restart.

---

Another workaround I tried, was to mount /proc/sys from the host inside chroot_env but even that had no effect:

      # Kernel Params
      "/proc/sys/fs/file-max" = "/proc/sys/fs/file-max"
      "/proc/sys/net/ipv4/ip_local_port_range" = "/proc/sys/net/ipv4/ip_local_port_range"
      "/proc/sys/net/netfilter/nf_conntrack_max" = "/proc/sys/net/netfilter/nf_conntrack_max" 
      "/proc/sys/net/ipv4/tcp_syncookies" = "/proc/sys/net/ipv4/tcp_syncookies"
      "/proc/sys/net/core/somaxconn" = "/proc/sys/net/core/somaxconn" 
      "/proc/sys/net/ipv4/tcp_max_syn_backlog" = "/proc/sys/net/ipv4/tcp_max_syn_backlog"
      "/proc/sys/net/ipv4/tcp_slow_start_after_idle" = "/proc/sys/net/ipv4/tcp_slow_start_after_idle"
      "/proc/sys/net/ipv4/tcp_fin_timeout" = "/proc/sys/net/ipv4/tcp_fin_timeout"
      "/proc/sys/net/ipv4/tcp_keepalive_time" = "/proc/sys/net/ipv4/tcp_keepalive_time"
      "/proc/sys/vm/overcommit_memory" = "/proc/sys/vm/overcommit_memory"

Is there any way to signal "exec" driver to load sysctl kernel params before the task gets started? Seems like a huge bummer for deploying high throughput apps on exec.

[tgross]

👋 I'm going to close out this issue because frankly, we're not likely to implement limits in the exec driver at this point without a full refresh of the driver which we've been chatting about internally. When we open an issue around an exec driver refresh, we'll be sure to keep this problem in mind.

@mr-karan, as for what you're seeing note that Nomad's exec driver doesn't set limits at all: that's actually what this ancient feature request is all about! The limits are being set elsewhere on your system, usually inherited the systemd service unit. I suspect what's happening here is that the executor is getting reparented to PID1 (systemd) when you're restarting the Nomad client, and then systemd is applying the default limits there. But if you dig into that some more and need more help, please feel free to open a new issue.

[prologic]

Hmmm it's not that hard to write the code to exec the process in a new cgroups and apply some of hte same limits you can apply in other types of Nomad jobs.

[tgross]

It's not a matter of difficulty, it's a matter of piling more features onto the exec driver in its current form, which we're not intending to do.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.