drivers/exec+java: reduce default set of linux capabilities #10600

shoenig · 2021-05-15T23:40:07Z

d817d12

drivers/exec: enable setting allow_caps on exec driver

This PR enables setting allow_caps on the exec driver
plugin configuration, as well as cap_add and cap_drop in
exec task configuration. These options replicate the
functionality already present in the docker task driver.

Important: this change also reduces the default set of
capabilities enabled by the exec driver to match the
default set enabled by the docker driver. Until v1.0.5
the exec task driver would enable all capabilities supported
by the operating system. v1.0.5 removed NET_RAW from that
list of default capabilities, but left may others which
could potentially also be leveraged by compromised tasks.

Important: the "root" user is still special cased when
used with the exec driver. Older versions of Nomad enabled
enabled all capabilities supported by the operating system
for tasks set with the root user. To maintain compatibility
with existing clusters we continue supporting this "feature",
however we maintain support for the legacy set of capabilities
rather than enabling all capabilities now supported on modern
operating systems.

c53a948

drivers/java: enable setting allow_caps on java driver

Enable setting allow_caps on the java task driver plugin, along
with the associated cap_add and cap_drop options in java task
configuration.

798418295

drivers/docker: reuse capabilities plumbing in docker driver

This changeset does not introduce any functional change for the
docker driver, but rather cleans up the implementation around
computing configured capabilities by re-using code written for
the exec/java task drivers.

a71081c9b

docs: update docs for linux capabilities in exec/java/docker drivers

Update docs for allow_caps, cap_add, cap_drop in exec/java/docker driver
pages. Also update upgrade guide with guidance on new default linux
capabilities for exec and java drivers.

notnoop

The changes and the code LGTM.

I'm uncertain about the timing - whether we have sufficient baking time for 1.1.0 or whether to wait until 1.2.0. While it's backward incompatible, we have plenty of precedence for security related backward incompatible changes in point releases at this point.

notnoop · 2021-05-17T12:59:58Z

drivers/shared/executor/executor.go

+//
+// Important (!): when adding fields, make sure to update the RPC methods in
+// grpcExecutorClient.Launch and grpcExecutorServer.Launch. Number of hours
+// spent tracking this down: too many.


🤦 I've been bitten by this too; or when you update the serializer but not deserializer. For these types, it might make sense to have a property test that ensures data round-trips without any loss.

notnoop · 2021-05-17T13:14:38Z

drivers/shared/executor/executor_linux.go

+		// when running as root, use the legacy set of system capabilities, so
+		// that we do not break existing nomad clusters using this "feature"


This is unfortunate. The good thing is that allowing root in exec tasks is an opt-in behavior. We can also change it and provide the legacy configuration flag.

notnoop · 2021-05-17T13:21:24Z

drivers/shared/executor/executor_linux_test.go

@@ -494,7 +495,6 @@ CapAmb: 0000000000000000`,

 	for _, c := range cases {
 		t.Run(c.user, func(t *testing.T) {
-			require := require.New(t)


This delights me - I dislike require.New(t) - it doesn't really save any keystores and makes it easy to get subtest assertions associated with the wrong t.

notnoop · 2021-05-17T13:32:44Z

drivers/shared/capabilities/set.go

+// upper - indicates whether to uppercase and prefix capabilities with CAP_
+func (s *Set) Slice(upper bool) []string {


Do these representations (caps_ upper case vs "naked" lower case) have a proper names or distinct usage? It might make sense to have different methods with distinct names? The boolean parameter is hard to decipher on the call sites.

This is what gets called as input to either docker (lower-case) or libcontainer (upper-case). Log messages and tests use lower case, just because (IMHO) it's easier on the eyes.

We could have two methods - maybe Enums() and ... Args()? Naming the second one is difficult 🙂

picatz · 2021-05-17T14:03:55Z

website/content/docs/upgrade/upgrade-specific.mdx

+SYS_TIME  SYS_TTY_CONFIG  WAKE_ALARM
+```
+
+The capabilities now enabled by default are modeled after Docker default [`linux capabilities`]:


I think this line might be a bit misleading/confusing:

linux capabilities doesn't seem to link anywhere, and probably needs to?

the capabilities listed below don't exactly match what Nomad uses, because it includes NET_RAW. Might be worth re-clarifying this to make it clear to administrators performing an upgrade what capabilities their tasks will have.

The anchor magic at the bottom links linux capabilities to https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities (the interesting bit of which is the table that can't be linked directly to)

Ahhhh nice catch! I copied this list from the old set; the new set does not include NET_RAW

picatz · 2021-05-17T14:05:45Z

website/content/docs/drivers/java.mdx

+```
+
+  which is modeled after the capabilities allowed by [docker by default][docker_caps]
+  (sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities


😅 Honestly, I didn't know what "sans" meant. Might be worth considering if this should be something simpler, like "without"?

Suggested change

(sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities

(without [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities

😬 You aren't the only - I get confused every time I try to remember the meaning of San Serif fonts :).

This PR enables setting allow_caps on the exec driver plugin configuration, as well as cap_add and cap_drop in exec task configuration. These options replicate the functionality already present in the docker task driver. Important: this change also reduces the default set of capabilities enabled by the exec driver to match the default set enabled by the docker driver. Until v1.0.5 the exec task driver would enable all capabilities supported by the operating system. v1.0.5 removed NET_RAW from that list of default capabilities, but left may others which could potentially also be leveraged by compromised tasks. Important: the "root" user is still special cased when used with the exec driver. Older versions of Nomad enabled enabled all capabilities supported by the operating system for tasks set with the root user. To maintain compatibility with existing clusters we continue supporting this "feature", however we maintain support for the legacy set of capabilities rather than enabling all capabilities now supported on modern operating systems.

Enable setting allow_caps on the java task driver plugin, along with the associated cap_add and cap_drop options in java task configuration.

This changeset does not introduce any functional change for the docker driver, but rather cleans up the implementation around computing configured capabilities by re-using code written for the exec/java task drivers.

Update docs for allow_caps, cap_add, cap_drop in exec/java/docker driver pages. Also update upgrade guide with guidance on new default linux capabilities for exec and java drivers.

The error output being checked depends on the linux caps supported by the particular operating system. Fix these test cases to just check that an error did occur.

Looks like we no longer need a package.

Add capabilities to the LaunchRequest proto so that the capabilities set actually gets plumbed all the way through to task launch.

drivers/exec+java: reduce default set of linux capabilities

github-actions · 2022-11-22T02:30:53Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

vercel bot temporarily deployed to Preview – nomad May 15, 2021 23:56 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui May 15, 2021 23:56 View deployment

shoenig force-pushed the f-exec-allow_caps branch from dca4acd to cce5e47 Compare May 16, 2021 00:21

vercel bot deployed to Preview – nomad-storybook-and-ui May 16, 2021 00:21 View deployment

vercel bot temporarily deployed to Preview – nomad May 16, 2021 00:21 Inactive

vercel bot temporarily deployed to Preview – nomad May 16, 2021 00:22 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui May 16, 2021 00:23 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui May 16, 2021 16:08 View deployment

vercel bot temporarily deployed to Preview – nomad May 16, 2021 16:08 Inactive

shoenig marked this pull request as ready for review May 16, 2021 18:06

shoenig requested review from tgross and nickethier May 16, 2021 18:06

notnoop approved these changes May 17, 2021

View reviewed changes

picatz reviewed May 17, 2021

View reviewed changes

shoenig added 8 commits May 17, 2021 12:37

drivers/java: enable setting allow_caps on java driver

9bb4b8f

Enable setting allow_caps on the java task driver plugin, along with the associated cap_add and cap_drop options in java task configuration.

drivers/docker: reuse capabilities plumbing in docker driver

c34beb4

This changeset does not introduce any functional change for the docker driver, but rather cleans up the implementation around computing configured capabilities by re-using code written for the exec/java task drivers.

docs: update docs for linux capabilities in exec/java/docker drivers

7245ac3

Update docs for allow_caps, cap_add, cap_drop in exec/java/docker driver pages. Also update upgrade guide with guidance on new default linux capabilities for exec and java drivers.

drivers: fixup linux version dependent test cases

17ec5a5

The error output being checked depends on the linux caps supported by the particular operating system. Fix these test cases to just check that an error did occur.

deps: update go mod tidy

683751d

Looks like we no longer need a package.

drivers/exec: pass capabilities through executor RPC

595cef8

Add capabilities to the LaunchRequest proto so that the capabilities set actually gets plumbed all the way through to task launch.

docs: minor wording tweaks + cl

845a3d3

shoenig force-pushed the f-exec-allow_caps branch from 1b382f5 to 845a3d3 Compare May 17, 2021 18:53

vercel bot deployed to Preview – nomad-storybook-and-ui May 17, 2021 18:53 View deployment

vercel bot deployed to Preview – nomad May 17, 2021 18:53 View deployment

shoenig merged commit 3f34f93 into main May 17, 2021

shoenig deleted the f-exec-allow_caps branch May 17, 2021 19:11

tgross pushed a commit that referenced this pull request May 17, 2021

Merge pull request #10600 from hashicorp/f-exec-allow_caps

2f924a9

drivers/exec+java: reduce default set of linux capabilities

tgross mentioned this pull request May 18, 2021

proposal: reduced default Linux capabilities #10611

Open

github-actions bot locked as resolved and limited conversation to collaborators Nov 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drivers/exec+java: reduce default set of linux capabilities #10600

drivers/exec+java: reduce default set of linux capabilities #10600

shoenig commented May 15, 2021

notnoop left a comment

notnoop May 17, 2021

notnoop May 17, 2021

notnoop May 17, 2021

notnoop May 17, 2021

shoenig May 17, 2021

picatz May 17, 2021

shoenig May 17, 2021

picatz May 17, 2021

notnoop May 17, 2021

github-actions bot commented Nov 22, 2022

		// when running as root, use the legacy set of system capabilities, so
		// that we do not break existing nomad clusters using this "feature"

		// upper - indicates whether to uppercase and prefix capabilities with CAP_
		func (s *Set) Slice(upper bool) []string {

	(sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities
	(without [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities

drivers/exec+java: reduce default set of linux capabilities #10600

drivers/exec+java: reduce default set of linux capabilities #10600

Conversation

shoenig commented May 15, 2021

notnoop left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 22, 2022