client: fix interpolation in template source #9383

schmichael · 2020-11-18T01:25:30Z

Followup to #9149

While Nomad v0.12.8 fixed NOMAD_{ALLOC,TASK,SECRETS}_DIR use in
template.destination, interpolating these variables in
template.source caused a path escape error.

repro

Why not apply the destination fix to source?

The destination fix forces destination to always be relative to the task
directory. This makes sense for the destination as a destination outside
the task directory would be unreachable by the task. There's no reason
to ever render a template outside the task directory. (Using .. does
allow destinations to escape the task directory if
template.disable_file_sandbox = true. That's just awkward and unsafe
enough I hope no one uses it.)

There is a reason to source a template outside a task
directory. At least if there weren't then I can't think of why we
implemented template.disable_file_sandbox. So v0.12.8 left the
behavior of template.source the more straightforward "Interpolate and
validate."

However, since outside of raw_exec every other driver uses absolute
paths for NOMAD_*_DIR interpolation, this means those variables are
unusable unless disable_file_sandbox is set.

The Fix

The variables are now interpolated as relative paths only for the
purpose of rendering templates. This is an unfortunate special case,
but reflects the fact that the templates view of the filesystem is
completely different (unconstrainted) vs the task's view (chrooted).
Arguably the values of these variables should be context-specific.
I think it's more reasonable to think of the "hack" as templating
running uncontainerized than that giving templates different paths is a
hack.

raw_exec

Filed: #9389

raw_exec is actually broken a different way as exercised by tests in
this commit. I think we should probably remove these tests and fix that
in a followup PR/release, but I wanted to leave them in for the initial
review and discussion. Since non-containerized source paths are broken
anyway, perhaps there's another solution to this entire problem I'm
overlooking?

While Nomad v0.12.8 fixed `NOMAD_{ALLOC,TASK,SECRETS}_DIR` use in `template.destination`, interpolating these variables in `template.source` caused a path escape error. **Why not apply the destination fix to source?** The destination fix forces destination to always be relative to the task directory. This makes sense for the destination as a destination outside the task directory would be unreachable by the task. There's no reason to ever render a template outside the task directory. (Using `..` does allow destinations to escape the task directory if `template.disable_file_sandbox = true`. That's just awkward and unsafe enough I hope no one uses it.) There is a reason to source a template outside a task directory. At least if there weren't then I can't think of why we implemented `template.disable_file_sandbox`. So v0.12.8 left the behavior of `template.source` the more straightforward "Interpolate and validate." However, since outside of `raw_exec` every other driver uses absolute paths for `NOMAD_*_DIR` interpolation, this means those variables are unusable unless `disable_file_sandbox` is set. **The Fix** The variables are now interpolated as relative paths *only for the purpose of rendering templates.* This is an unfortunate special case, but reflects the fact that the templates view of the filesystem is completely different (unconstrainted) vs the task's view (chrooted). Arguably the values of these variables *should be context-specific.* I think it's more reasonable to think of the "hack" as templating running uncontainerized than that giving templates different paths is a hack. **TODO** - [ ] E2E tests - [ ] Job validation may still be broken and prevent my fix from working? **raw_exec** `raw_exec` is actually broken _a different way_ as exercised by tests in this commit. I think we should probably remove these tests and fix that in a followup PR/release, but I wanted to leave them in for the initial review and discussion. Since non-containerized source paths are broken anyway, perhaps there's another solution to this entire problem I'm overlooking?

tgross

The patch itself LGTM. IIRC we considered this approach for 0.12.7 but I don't remember why we decided not to go that way. Definitely some open questions about how raw_exec is supposed to be behaving.

tgross · 2020-11-18T13:19:30Z

client/allocrunner/taskrunner/template/template_test.go

@@ -1452,6 +1454,250 @@ func TestTaskTemplateManager_Config_VaultNamespace_TaskOverride(t *testing.T) {
 	assert.Equal(overriddenNS, *ctconf.Vault.Namespace, "Vault Namespace Value")
 }

+// TestTaskTemplateManager_Escapes asserts that when sandboxing is enabled
+// interpolated paths are not incorrectly treated as escaping the alloc dir.
+func TestTaskTemplateManager_Escapes(t *testing.T) {


There's an existing test at https://github.com/hashicorp/nomad/blob/v0.12.8/client/allocrunner/taskrunner/template/template_test.go#L362-L452 that covers using interpolation to escape. This test setup is nicer so maybe we could move those test cases into this one?

I would prefer to leave that test be for now. It's definitely uglier, but it's a bit more comprehensive than this minimal unit test. There's so many state permutations to test in templates I'm pretty nervous reworking that test might miss some subtle state (eg actually executing the templates and meta var interpolation are unique to that test).

client/allocrunner/taskrunner/template/template_test.go

tgross · 2020-11-18T13:56:09Z

client/allocrunner/taskrunner/template/template_test.go

+		},
+		//TODO: Fix this test. I *think* it should pass. The double
+		//      joining of the task dir onto the destination seems like
+		//      a bug.


The regression patch in 0.12.7 (the CVE fix was 0.12.6) made it so that we always append the destination onto the task directory (ref).

So I checked out this PR and removed the patch code, and it looks like this test fails in the same way. This either an existing bug or a regression that 0.12.7 introduced. 😦

=== RUN TestTaskTemplateManager_Escapes/RawExecOk template_test.go:1688: Error Trace: template_test.go:1688 Error: Not equal: expected: "/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/secrets/dst" actual : "/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/secrets/dst" Diff: --- Expected +++ Actual @@ -1 +1 @@ -/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/secrets/dst +/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/fake/allocdir/8b3aaa2a-beca-ae55-2ece-304012490a0d/web/secrets/dst Test: TestTaskTemplateManager_Escapes/RawExecOk

Ok, I ported this test over to 0.12.5 (pre-CVE patch) and it still fails in the same way, so this is an existing bug. I'd propose we open a new issue for that and then comment out this test with a link to that issue, so we remember to resolve it later and not hold this up.

=== RUN TestTaskTemplateManager_Escapes/RawExecOk template_test.go:1749: Error Trace: template_test.go:1749 Error: Not equal: expected: "/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/local/src" actual : "/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/local/src" Diff: --- Expected +++ Actual @@ -1 +1 @@ -/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/local/src +/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/fake/allocdir/254b7cac-88fa-b795-da67-8e28cc507ec7/web/local/src Test: TestTaskTemplateManager_Escapes/RawExecOk

Thanks for testing. Issue filed: #9389

Going to skip these for now and reference that issue.

tgross · 2020-11-18T15:01:00Z

The E2E test consultemplate doesn't exercise the source field at all, unfortunately. Otherwise we might have caught this one. (It also doesn't exercise raw_exec.) Maybe we could place a source that's in one of NOMAD_{ALLOC,TASK,SECRETS}_DIR by writing it in a pre-start task?

tgross · 2020-11-18T20:07:21Z

E2E test LGTM.

Ran these tests on a full E2E cluster built from this sha:

=== RUN   TestE2E
...
=== RUN   TestE2E/ConsulTemplate
=== RUN   TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest
=== RUN   TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplatePathInterpolation_Bad
=== RUN   TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplatePathInterpolation_Ok
=== RUN   TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplateUpdateTriggers
...
=== RUN   TestE2E/NodeDrain
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainDeadline
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainEphemeralMigrate
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainForce
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainIgnoreSystem
=== RUN   TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainKeepIneligible
...
=== RUN   TestE2E/VaultSecrets
=== RUN   TestE2E/VaultSecrets/*vaultsecrets.VaultSecretsTest
=== RUN   TestE2E/VaultSecrets/*vaultsecrets.VaultSecretsTest/TestVaultSecrets
...
--- PASS: TestE2E (49.94s)
    --- PASS: TestE2E/ConsulTemplate (49.94s)
        --- PASS: TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest (49.94s)
            --- PASS: TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplatePathInterpolation_Bad (2.52s)
            --- PASS: TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplatePathInterpolation_Ok (4.67s)
            --- PASS: TestE2E/ConsulTemplate/*consultemplate.ConsulTemplateTest/TestTemplateUpdateTriggers (42.41s)
...
    --- PASS: TestE2E/NodeDrain (117.30s)
        --- PASS: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest (117.30s)
            --- FAIL: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainDeadline (36.53s)
            --- PASS: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainEphemeralMigrate (23.51s)
            --- PASS: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainForce (34.23s)
            --- PASS: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainIgnoreSystem (22.22s)
            --- PASS: TestE2E/NodeDrain/*nodedrain.NodeDrainE2ETest/TestNodeDrainKeepIneligible (0.61s)
...
    --- PASS: TestE2E/VaultSecrets (71.66s)
        --- PASS: TestE2E/VaultSecrets/*vaultsecrets.VaultSecretsTest (71.65s)
            --- PASS: TestE2E/VaultSecrets/*vaultsecrets.VaultSecretsTest/TestVaultSecrets (71.45s)

Backport of client: fix interpolation in template source #9383

Backport to 0.11 of client: fix interpolation in template source #9383

Backport to 0.10 of client: fix interpolation in template source #9383

upstream fix: #9383 backport PRs: #9391, #9402, and #9405

github-actions · 2022-12-10T02:16:49Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

schmichael changed the title ~~[WIP~~ [WIP] Fix template.source validation Nov 18, 2020

schmichael force-pushed the b-template-escape branch from 24def15 to cd7226d Compare November 18, 2020 06:19

vercel bot temporarily deployed to Preview November 18, 2020 06:19 Inactive

schmichael changed the title ~~[WIP] Fix template.source validation~~ [WIP] client: fix interpolation in template source Nov 18, 2020

schmichael requested review from tgross and notnoop November 18, 2020 06:20

tgross approved these changes Nov 18, 2020

View reviewed changes

client: skip broken test and fix assertion

5b83ca0

vercel bot temporarily deployed to Preview November 18, 2020 18:01 Inactive

vercel bot temporarily deployed to Preview November 18, 2020 18:44 Inactive

schmichael changed the title ~~[WIP] client: fix interpolation in template source~~ client: fix interpolation in template source Nov 18, 2020

schmichael marked this pull request as ready for review November 18, 2020 18:45

e2e: test template path interpolation

252ac1e

schmichael force-pushed the b-template-escape branch from fe91adc to 252ac1e Compare November 18, 2020 18:49

vercel bot temporarily deployed to Preview November 18, 2020 18:49 Inactive

schmichael merged commit 292e7a4 into master Nov 18, 2020

schmichael deleted the b-template-escape branch November 18, 2020 20:21

schmichael mentioned this pull request Nov 18, 2020

Backport of client: fix interpolation in template source #9383 #9391

Merged

schmichael added a commit that referenced this pull request Nov 18, 2020

Merge pull request #9391 from hashicorp/b-0129-backport

9a838b8

Backport of client: fix interpolation in template source #9383

schmichael mentioned this pull request Nov 19, 2020

Backport to 0.11 of client: fix interpolation in template source #9383 #9402

Merged

schmichael added a commit that referenced this pull request Nov 19, 2020

Merge pull request #9402 from hashicorp/b-0118-backport

013a195

Backport to 0.11 of client: fix interpolation in template source #9383

schmichael mentioned this pull request Nov 19, 2020

Backport to 0.10 of client: fix interpolation in template source #9383 #9405

Merged

schmichael added a commit that referenced this pull request Nov 19, 2020

Merge pull request #9405 from hashicorp/b-0109-backport

0267ec7

Backport to 0.10 of client: fix interpolation in template source #9383

schmichael added a commit that referenced this pull request Nov 19, 2020

docs: add 0.12.9, 0.11.8, and 0.10.9 to changelog

9f7c134

upstream fix: #9383 backport PRs: #9391, #9402, and #9405

cgbaker mentioned this pull request Dec 22, 2020

address interpolation of artifact and template src/dest #9671

Merged

github-actions bot locked as resolved and limited conversation to collaborators Dec 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: fix interpolation in template source #9383

client: fix interpolation in template source #9383

schmichael commented Nov 18, 2020 •

edited

Loading

tgross left a comment •

edited

Loading

tgross Nov 18, 2020 •

edited

Loading

schmichael Nov 18, 2020

tgross Nov 18, 2020

tgross Nov 18, 2020 •

edited

Loading

schmichael Nov 18, 2020

tgross commented Nov 18, 2020 •

edited

Loading

tgross commented Nov 18, 2020 •

edited

Loading

github-actions bot commented Dec 10, 2022

client: fix interpolation in template source #9383

client: fix interpolation in template source #9383

Conversation

schmichael commented Nov 18, 2020 • edited Loading

tgross left a comment • edited Loading

Choose a reason for hiding this comment

tgross Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

schmichael Nov 18, 2020

Choose a reason for hiding this comment

tgross Nov 18, 2020

Choose a reason for hiding this comment

tgross Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

schmichael Nov 18, 2020

Choose a reason for hiding this comment

tgross commented Nov 18, 2020 • edited Loading

tgross commented Nov 18, 2020 • edited Loading

github-actions bot commented Dec 10, 2022

schmichael commented Nov 18, 2020 •

edited

Loading

tgross left a comment •

edited

Loading

tgross Nov 18, 2020 •

edited

Loading

tgross Nov 18, 2020 •

edited

Loading

tgross commented Nov 18, 2020 •

edited

Loading

tgross commented Nov 18, 2020 •

edited

Loading