[Fleet] Encrypt ssl fields in logstash output #129131

nchaulet · 2022-03-31T19:15:28Z

Summary

Encrypt the ssl secrets in the logstash output using encrypted saved object
This require the user to set the encryption key in kibana.yml

UI Changes

joshdover · 2022-04-01T12:45:00Z

Yeah I was afraid of this possibility. The right way to solve this would be to add a parameter to the output service to accept a regular SO client, with the encrypted SO client wrapper. This would requiring adding a new parameter to several code paths for this client, including any path that modifies an agent policy or integration policy since those result in needing to generating the full agent policy in the .fleet-policies index.

As discussed on Slack, I think we'll need to do something similar to this fake request strategy used here: https://github.com/elastic/kibana/blob/8bad83e3eb5d03c5dfbaf18069c707f8a87c0fa3/x-pack/plugins/fleet/server/services/agent_policy_update.ts/#L39

nchaulet · 2022-04-01T13:32:24Z

@joshdover I was able to get it working with the fake request hack, I had to wait on the licence to be available otherwise I get an error with unauthorized call to .kibana.
If it's the fix we want to use I can come with a fully working PR today.

x-pack/plugins/fleet/server/plugin.ts

nchaulet · 2022-04-01T15:47:24Z

@elasticmachine merge upstream

elasticmachine · 2022-04-01T18:30:39Z

Pinging @elastic/fleet (Team:Fleet)

...ublic/applications/fleet/sections/settings/components/edit_output_flyout/use_output_form.tsx

...ns/fleet/sections/settings/components/edit_output_flyout/encryption_key_required_callout.tsx

x-pack/plugins/fleet/server/routes/setup/handlers.ts

joshdover

Overall approach LGTM

x-pack/plugins/fleet/server/services/output.ts

joshdover · 2022-04-04T10:43:20Z

...ns/fleet/sections/settings/components/edit_output_flyout/encryption_key_required_callout.tsx

+    >
+      <FormattedMessage
+        id="xpack.fleet.encryptionKeyRequired.calloutDescription"
+        defaultMessage="To configure logstash output, set a value of {key} in your {file} file. {link}"


nit: I wonder if we should use different wording here since not all environments use kibana.yml (eg. docker users use env vars often).

yes we probably can update this, (I think it can come as a follow up PR as merging this one will unlock testing the logstash output) the doc do not specify the docker env var variables either maybe it could be fixed here.

nchaulet · 2022-04-04T12:23:48Z

@elasticmachine merge upstream

dedemorton

I've offered suggestions for text string changes, but would like @gchaps to confirm.

...ns/fleet/sections/settings/components/edit_output_flyout/encryption_key_required_callout.tsx

...eet/public/applications/fleet/sections/settings/components/edit_output_flyout/index.test.tsx

x-pack/plugins/fleet/server/services/output.test.ts

Co-authored-by: DeDe Morton <[email protected]>

hop-dev · 2022-04-05T13:41:17Z

x-pack/plugins/fleet/server/services/output.ts

-      id: options?.id ?? newSo.id,
-      ...newSo.attributes,
-    };
+    return outputSavedObjectToOutput(newSo);


So is newSo.ssl unencrypted here? (but has been encrypted when written to ES?)

Yes that is correct, when you use dangerouslyExposeValue in the saved object declaration the wrapped saved object client return the decrypted values

nchaulet · 2022-04-05T14:45:06Z

@elasticmachine merge upstream

…ibana into encrypt-saved-object-logstash

dedemorton

UI text LGTM!

gchaps

UI copy and docslink LGTM.

nchaulet · 2022-04-05T18:01:44Z

@elasticmachine merge upstream

kibana-ci · 2022-04-05T19:30:37Z

💛 Build succeeded, but was flaky

Test Failures

[job] [logs] Default CI Group #12 / alerting api integration spaces only Alerting builtin alertTypes es_query alert runs correctly: use epoch millis - threshold on hit count < > for searchSource search type

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	578	579	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	683.2KB	684.2KB	+992.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`core`	318.7KB	318.7KB	+65.0B

History

💚 Build #36227 succeeded 5d572d5
💔 Build #36196 failed ad77343
💔 Build #36193 failed c4f38f3
💔 Build #36166 failed 7370297
💚 Build #35863 succeeded bb1b463
💚 Build #35759 succeeded ec9b8f7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

(cherry picked from commit 420359b)

kibanamachine · 2022-04-05T19:34:23Z

💚 All backports created successfully

Status	Branch	Result
✅	8.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit 420359b) Co-authored-by: Nicolas Chaulet <[email protected]>

Test encrypt ssl fleet server

bf5701a

Use a fake request to get a scoped client

71dc3bf

joshdover reviewed Apr 1, 2022

View reviewed changes

x-pack/plugins/fleet/server/plugin.ts Outdated Show resolved Hide resolved

nchaulet added 3 commits April 1, 2022 10:01

Fix types

276ec97

Use encrypted so client only for output

7b656a2

Fix typo

c7c6f10

kibanamachine and others added 2 commits April 1, 2022 11:47

Merge branch 'main' into encrypt-saved-object-logstash

8943f28

Add UI if encrypted saved object is not set

8c021e0

nchaulet self-assigned this Apr 1, 2022

nchaulet added release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v8.2.0 v8.3.0 labels Apr 1, 2022

nchaulet requested a review from joshdover April 1, 2022 17:38

fix i18n key

77656a6

nchaulet changed the title ~~Test encrypt ssl fleet server~~ [Fleet] Encrypt ssl fields in logstash output Apr 1, 2022

nchaulet marked this pull request as ready for review April 1, 2022 17:42

nchaulet requested a review from a team as a code owner April 1, 2022 17:42

Fix i18n

e1cb83c

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 1, 2022

jen-huang reviewed Apr 1, 2022

View reviewed changes

...ublic/applications/fleet/sections/settings/components/edit_output_flyout/use_output_form.tsx Show resolved Hide resolved

jen-huang reviewed Apr 1, 2022

View reviewed changes

Fix after review

823ab42

nchaulet requested a review from a team as a code owner April 1, 2022 20:08

Fix create

ec9b8f7

joshdover approved these changes Apr 4, 2022

View reviewed changes

Merge branch 'main' into encrypt-saved-object-logstash

bb1b463

dedemorton reviewed Apr 5, 2022

View reviewed changes

nchaulet and others added 2 commits April 5, 2022 08:55

Apply suggestions from code review

1b50505

Co-authored-by: DeDe Morton <[email protected]>

Update copy

7370297

hop-dev reviewed Apr 5, 2022

View reviewed changes

fix i18n

1fa7245

kibanamachine and others added 3 commits April 5, 2022 10:45

Merge branch 'main' into encrypt-saved-object-logstash

c4f38f3

Fix unused

ecc19ed

Merge branch 'encrypt-saved-object-logstash' of github.com:nchaulet/k…

ad77343

…ibana into encrypt-saved-object-logstash

nchaulet requested review from gchaps and dedemorton April 5, 2022 15:43

Fix jest test

5d572d5

dedemorton approved these changes Apr 5, 2022

View reviewed changes

gchaps approved these changes Apr 5, 2022

View reviewed changes

Merge branch 'main' into encrypt-saved-object-logstash

cdc0020

nchaulet enabled auto-merge (squash) April 5, 2022 18:53

nchaulet merged commit 420359b into elastic:main Apr 5, 2022

kibanamachine mentioned this pull request Apr 5, 2022

[8.2] [Fleet] Encrypt ssl fields in logstash output (#129131) #129525

Merged

kibanamachine pushed a commit that referenced this pull request Apr 5, 2022

[Fleet] Encrypt ssl fields in logstash output (#129131)

86e3304

(cherry picked from commit 420359b)

kibanamachine added a commit that referenced this pull request Apr 5, 2022

[Fleet] Encrypt ssl fields in logstash output (#129131) (#129525)

9bec00d

(cherry picked from commit 420359b) Co-authored-by: Nicolas Chaulet <[email protected]>

dedemorton added the ui-copy Review of UI copy with docs team is recommended label Apr 5, 2022

dedemorton mentioned this pull request Apr 6, 2022

Edit text strings and links in add logstash output UI #129551

Merged

9 tasks

nchaulet mentioned this pull request Apr 6, 2022

[Self-Managed]: Security is not enabled by default on setting up self-managed 8.2 Snapshot. #129568

Closed

dedemorton mentioned this pull request Apr 6, 2022

Mention other ways to set encryption key for saved objects #129671

Closed

joshdover mentioned this pull request Jun 10, 2022

[Fleet] Output preconfiguration can break on re-authentication attempts #134165

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Encrypt ssl fields in logstash output #129131

[Fleet] Encrypt ssl fields in logstash output #129131

nchaulet commented Mar 31, 2022 •

edited

Loading

joshdover commented Apr 1, 2022

nchaulet commented Apr 1, 2022

nchaulet commented Apr 1, 2022

elasticmachine commented Apr 1, 2022

joshdover left a comment

joshdover Apr 4, 2022

nchaulet Apr 4, 2022

nchaulet commented Apr 4, 2022

dedemorton left a comment

hop-dev Apr 5, 2022

nchaulet Apr 5, 2022

nchaulet commented Apr 5, 2022

dedemorton left a comment

gchaps left a comment

nchaulet commented Apr 5, 2022

kibana-ci commented Apr 5, 2022

kibanamachine commented Apr 5, 2022

[Fleet] Encrypt ssl fields in logstash output #129131

[Fleet] Encrypt ssl fields in logstash output #129131

Conversation

nchaulet commented Mar 31, 2022 • edited Loading

Summary

UI Changes

joshdover commented Apr 1, 2022

nchaulet commented Apr 1, 2022

nchaulet commented Apr 1, 2022

elasticmachine commented Apr 1, 2022

joshdover left a comment

Choose a reason for hiding this comment

joshdover Apr 4, 2022

Choose a reason for hiding this comment

nchaulet Apr 4, 2022

Choose a reason for hiding this comment

nchaulet commented Apr 4, 2022

dedemorton left a comment

Choose a reason for hiding this comment

hop-dev Apr 5, 2022

Choose a reason for hiding this comment

nchaulet Apr 5, 2022

Choose a reason for hiding this comment

nchaulet commented Apr 5, 2022

dedemorton left a comment

Choose a reason for hiding this comment

gchaps left a comment

Choose a reason for hiding this comment

nchaulet commented Apr 5, 2022

kibana-ci commented Apr 5, 2022

💛 Build succeeded, but was flaky

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

kibanamachine commented Apr 5, 2022

💚 All backports created successfully

Questions ?

nchaulet commented Mar 31, 2022 •

edited

Loading