Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use kibana_system user for Fleet setup and package operations #112808

Merged
merged 12 commits into from
Oct 15, 2021

Conversation

joshdover
Copy link
Contributor

@joshdover joshdover commented Sep 22, 2021

Summary

Fixes #111755
Blocked by:

Uses the kibana_system user under the hood for executing all package installation operations. No external behavior changes are expected from this change.

Because transforms require read and create_index privileges and we can't give kibana_system access to all indices, we no longer support arbitrary transforms. They specifically need privileges granted to kibana_system in the Elasticsearch codebase to work. Therefore, the transforms in our test package no longer work. I've removed these transforms and added tests specific to the only package we currently support this in, endpoint.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@joshdover
Copy link
Contributor Author

The functional tests that are failing on this PR are blocked by our Elasticsearch snapshot being promoted to a more recent build. I've pinged the appropriate teams to make sure we can get that promoted ASAP. The job that verifies the snapshot promotion is here: https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify

Comment on lines +112 to +116
export const getFileHandler: FleetRequestHandler<TypeOf<typeof GetFileRequestSchema.params>> =
async (context, request, response) => {
try {
const { pkgName, pkgVersion, filePath } = request.params;
const savedObjectsClient = context.fleet.epm.internalSoClient;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only changes on this function are the change to FleetRequestHandler type and using internalSoClient instead of the current user one. Everything else is the same, but got re-formatted by the linter.

Comment on lines +186 to +190
export const getInfoHandler: FleetRequestHandler<TypeOf<typeof GetInfoRequestSchema.params>> =
async (context, request, response) => {
try {
const { pkgkey } = request.params;
const savedObjectsClient = context.fleet.epm.internalSoClient;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as well

@joshdover joshdover added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.16.0 v8.0.0 labels Oct 11, 2021
@joshdover
Copy link
Contributor Author

Test failures are waiting on the next Elasticsearch snapshot to be promoted, otherwise, this is ready for review

@joshdover joshdover marked this pull request as ready for review October 11, 2021 14:12
@joshdover joshdover requested a review from a team as a code owner October 11, 2021 14:12
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@joshdover
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes all look sound to me. Pulled down and ran through setup + a few basic operations and all looks good. 🚀

Copy link
Contributor

@criamico criamico left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the branch locally, did the fleet setup and installed some packages. LGTM to me 👍

@joshdover
Copy link
Contributor Author

CI will fail until elastic/elasticsearch#79076 is merged and Kibana's CI snapshot has been bumped

@joshdover joshdover requested a review from kobelb October 13, 2021 18:34
@joshdover
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_threat_matching·ts.detection engine api security and spaces enabled create_threat_matching tests with auditbeat data should be able to execute and get 10 signals when doing a specific query

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]     │
[00:00:00]       └-: detection engine api security and spaces enabled
[00:00:00]         └-> "before all" hook in "detection engine api security and spaces enabled"
[00:00:00]         └-: 
[00:00:00]           └-> "before all" hook in ""
[00:10:21]           └-: create_threat_matching
[00:10:21]             └-> "before all" hook in "create_threat_matching"
[00:10:33]             └-: tests with auditbeat data
[00:10:33]               └-> "before all" hook for "should be able to execute and get 10 signals when doing a specific query"
[00:10:33]               └-> "before all" hook for "should be able to execute and get 10 signals when doing a specific query"
[00:10:33]                 │ info [x-pack/test/functional/es_archives/auditbeat/hosts] Loading "mappings.json"
[00:10:33]                 │ info [x-pack/test/functional/es_archives/auditbeat/hosts] Loading "data.json.gz"
[00:10:33]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [auditbeat-8.0.0-2019.02.19-000001] creating index, cause [api], templates [], shards [1]/[1]
[00:10:33]                 │ info [x-pack/test/functional/es_archives/auditbeat/hosts] Created index "auditbeat-8.0.0-2019.02.19-000001"
[00:10:33]                 │ debg [x-pack/test/functional/es_archives/auditbeat/hosts] "auditbeat-8.0.0-2019.02.19-000001" settings {"index":{"lifecycle":{"name":"auditbeat-8.0.0","rollover_alias":"auditbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["tags","message","agent.version","agent.name","agent.type","agent.id","agent.ephemeral_id","client.address","client.mac","client.domain","client.geo.continent_name","client.geo.country_name","client.geo.region_name","client.geo.city_name","client.geo.country_iso_code","client.geo.region_iso_code","client.geo.name","cloud.provider","cloud.availability_zone","cloud.region","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.account.id","container.runtime","container.id","container.image.name","container.image.tag","container.name","destination.address","destination.mac","destination.domain","destination.geo.continent_name","destination.geo.country_name","destination.geo.region_name","destination.geo.city_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.name","ecs.version","error.id","error.message","error.code","event.id","event.kind","event.category","event.action","event.outcome","event.type","event.module","event.dataset","event.hash","event.timezone","file.path","file.target_path","file.extension","file.type","file.device","file.inode","file.uid","file.owner","file.gid","file.group","file.mode","group.id","group.name","host.hostname","host.name","host.id","host.mac","host.type","host.architecture","host.os.platform","host.os.name","host.os.full","host.os.family","host.os.version","host.os.kernel","host.geo.continent_name","host.geo.country_name","host.geo.region_name","host.geo.city_name","host.geo.country_iso_code","host.geo.region_iso_code","host.geo.name","http.request.method","http.request.body.content","http.request.referrer","http.response.body.content","http.version","log.level","network.name","network.type","network.iana_number","network.transport","network.application","network.protocol","network.direction","network.community_id","observer.mac","observer.hostname","observer.vendor","observer.version","observer.serial_number","observer.type","observer.os.platform","observer.os.name","observer.os.full","observer.os.family","observer.os.version","observer.os.kernel","observer.geo.continent_name","observer.geo.country_name","observer.geo.region_name","observer.geo.city_name","observer.geo.country_iso_code","observer.geo.region_iso_code","observer.geo.name","organization.name","organization.id","os.platform","os.name","os.full","os.family","os.version","os.kernel","process.name","process.args","process.executable","process.title","process.working_directory","server.address","server.mac","server.domain","server.geo.continent_name","server.geo.country_name","server.geo.region_name","server.geo.city_name","server.geo.country_iso_code","server.geo.region_iso_code","server.geo.name","service.id","service.name","service.type","service.state","service.version","service.ephemeral_id","source.address","source.mac","source.domain","source.geo.continent_name","source.geo.country_name","source.geo.region_name","source.geo.city_name","source.geo.country_iso_code","source.geo.region_iso_code","source.geo.name","url.original","url.full","url.scheme","url.domain","url.path","url.query","url.fragment","url.username","url.password","user.id","user.name","user.full_name","user.email","user.hash","user.group.id","user.group.name","user_agent.original","user_agent.name","user_agent.version","user_agent.device.name","user_agent.os.platform","user_agent.os.name","user_agent.os.full","user_agent.os.family","user_agent.os.version","user_agent.os.kernel","agent.hostname","error.type","cloud.project.id","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.container.name","kubernetes.container.image","file.origin","raw","file.selinux.user","file.selinux.role","file.selinux.domain","file.selinux.level","user.audit.id","user.audit.name","user.effective.id","user.effective.name","user.effective.group.id","user.effective.group.name","user.filesystem.id","user.filesystem.name","user.filesystem.group.id","user.filesystem.group.name","user.saved.id","user.saved.name","user.saved.group.id","user.saved.group.name","user.selinux.user","user.selinux.role","user.selinux.domain","user.selinux.level","user.selinux.category","source.path","destination.path","auditd.message_type","auditd.session","auditd.result","auditd.summary.actor.primary","auditd.summary.actor.secondary","auditd.summary.object.type","auditd.summary.object.primary","auditd.summary.object.secondary","auditd.summary.how","auditd.paths.inode","auditd.paths.dev","auditd.paths.obj_user","auditd.paths.obj_role","auditd.paths.obj_domain","auditd.paths.obj_level","auditd.paths.objtype","auditd.paths.ouid","auditd.paths.rdev","auditd.paths.nametype","auditd.paths.ogid","auditd.paths.item","auditd.paths.mode","auditd.paths.name","auditd.data.action","auditd.data.minor","auditd.data.acct","auditd.data.addr","auditd.data.cipher","auditd.data.id","auditd.data.entries","auditd.data.kind","auditd.data.ksize","auditd.data.spid","auditd.data.arch","auditd.data.argc","auditd.data.major","auditd.data.unit","auditd.data.table","auditd.data.terminal","auditd.data.grantors","auditd.data.direction","auditd.data.op","auditd.data.tty","auditd.data.syscall","auditd.data.data","auditd.data.family","auditd.data.mac","auditd.data.pfs","auditd.data.items","auditd.data.a0","auditd.data.a1","auditd.data.a2","auditd.data.a3","auditd.data.hostname","auditd.data.lport","auditd.data.rport","auditd.data.exit","auditd.data.fp","auditd.data.laddr","auditd.data.sport","auditd.data.capability","auditd.data.nargs","auditd.data.new-enabled","auditd.data.audit_backlog_limit","auditd.data.dir","auditd.data.cap_pe","auditd.data.model","auditd.data.new_pp","auditd.data.old-enabled","auditd.data.oauid","auditd.data.old","auditd.data.banners","auditd.data.feature","auditd.data.vm-ctx","auditd.data.opid","auditd.data.seperms","auditd.data.seresult","auditd.data.new-rng","auditd.data.old-net","auditd.data.sigev_signo","auditd.data.ino","auditd.data.old_enforcing","auditd.data.old-vcpu","auditd.data.range","auditd.data.res","auditd.data.added","auditd.data.fam","auditd.data.nlnk-pid","auditd.data.subj","auditd.data.a[0-3]","auditd.data.cgroup","auditd.data.kernel","auditd.data.ocomm","auditd.data.new-net","auditd.data.permissive","auditd.data.class","auditd.data.compat","auditd.data.fi","auditd.data.changed","auditd.data.msg","auditd.data.dport","auditd.data.new-seuser","auditd.data.invalid_context","auditd.data.dmac","auditd.data.ipx-net","auditd.data.iuid","auditd.data.macproto","auditd.data.obj","auditd.data.ipid","auditd.data.new-fs","auditd.data.vm-pid","auditd.data.cap_pi","auditd.data.old-auid","auditd.data.oses","auditd.data.fd","auditd.data.igid","auditd.data.new-disk","auditd.data.parent","auditd.data.len","auditd.data.oflag","auditd.data.uuid","auditd.data.code","auditd.data.nlnk-grp","auditd.data.cap_fp","auditd.data.new-mem","auditd.data.seperm","auditd.data.enforcing","auditd.data.new-chardev","auditd.data.old-rng","auditd.data.outif","auditd.data.cmd","auditd.data.hook","auditd.data.new-level","auditd.data.sauid","auditd.data.sig","auditd.data.audit_backlog_wait_time","auditd.data.printer","auditd.data.old-mem","auditd.data.perm","auditd.data.old_pi","auditd.data.state","auditd.data.format","auditd.data.new_gid","auditd.data.tcontext","auditd.data.maj","auditd.data.watch","auditd.data.device","auditd.data.grp","auditd.data.bool","auditd.data.icmp_type","auditd.data.new_lock","auditd.data.old_prom","auditd.data.acl","auditd.data.ip","auditd.data.new_pi","auditd.data.default-context","auditd.data.inode_gid","auditd.data.new-log_passwd","auditd.data.new_pe","auditd.data.selected-context","auditd.data.cap_fver","auditd.data.file","auditd.data.net","auditd.data.virt","auditd.data.cap_pp","auditd.data.old-range","auditd.data.resrc","auditd.data.new-range","auditd.data.obj_gid","auditd.data.proto","auditd.data.old-disk","auditd.data.audit_failure","auditd.data.inif","auditd.data.vm","auditd.data.flags","auditd.data.nlnk-fam","auditd.data.old-fs","auditd.data.old-ses","auditd.data.seqno","auditd.data.fver","auditd.data.qbytes","auditd.data.seuser","auditd.data.cap_fe","auditd.data.new-vcpu","auditd.data.old-level","auditd.data.old_pp","auditd.data.daddr","auditd.data.old-role","auditd.data.ioctlcmd","auditd.data.smac","auditd.data.apparmor","auditd.data.fe","auditd.data.perm_mask","auditd.data.ses","auditd.data.cap_fi","auditd.data.obj_uid","auditd.data.reason","auditd.data.list","auditd.data.old_lock","auditd.data.bus","auditd.data.old_pe","auditd.data.new-role","auditd.data.prom","auditd.data.uri","auditd.data.audit_enabled","auditd.data.old-log_passwd","auditd.data.old-seuser","auditd.data.per","auditd.data.scontext","auditd.data.tclass","auditd.data.ver","auditd.data.new","auditd.data.val","auditd.data.img-ctx","auditd.data.old-chardev","auditd.data.old_val","auditd.data.success","auditd.data.inode_uid","auditd.data.removed","auditd.data.socket.port","auditd.data.socket.saddr","auditd.data.socket.addr","auditd.data.socket.family","auditd.data.socket.path","geoip.continent_name","geoip.city_name","geoip.region_name","geoip.country_iso_code","hash.blake2b_256","hash.blake2b_384","hash.blake2b_512","hash.md5","hash.sha1","hash.sha224","hash.sha256","hash.sha384","hash.sha3_224","hash.sha3_256","hash.sha3_384","hash.sha3_512","hash.sha512","hash.sha512_224","hash.sha512_256","hash.xxh64","event.origin","user.entity_id","user.terminal","process.entity_id","socket.entity_id","system.audit.host.timezone.name","system.audit.host.hostname","system.audit.host.id","system.audit.host.architecture","system.audit.host.mac","system.audit.host.os.platform","system.audit.host.os.name","system.audit.host.os.family","system.audit.host.os.version","system.audit.host.os.kernel","system.audit.package.entity_id","system.audit.package.name","system.audit.package.version","system.audit.package.release","system.audit.package.arch","system.audit.package.license","system.audit.package.summary","system.audit.package.url","system.audit.user.name","system.audit.user.uid","system.audit.user.gid","system.audit.user.dir","system.audit.user.shell","system.audit.user.user_information","system.audit.user.password.type","fields.*"]},"refresh_interval":"5s"}}
[00:10:34]                 │ info [x-pack/test/functional/es_archives/auditbeat/hosts] Indexed 1751 docs into "auditbeat-8.0.0-2019.02.19-000001"
[00:10:34]                 │ info [x-pack/test/functional/es_archives/auditbeat/hosts] Indexed 1 docs into "winlogbeat-8.0.0-2019.02.19-000001"
[00:10:34]               └-> should be able to execute and get 10 signals when doing a specific query
[00:10:34]                 └-> "before each" hook: global before each for "should be able to execute and get 10 signals when doing a specific query"
[00:10:34]                 └-> "before each" hook for "should be able to execute and get 10 signals when doing a specific query"
[00:10:34]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [node-01] adding index lifecycle policy [.siem-signals-default]
[00:10:34]                   │ info [o.e.c.m.MetadataIndexTemplateService] [node-01] adding index template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:10:34]                   │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1]
[00:10:34]                   │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:10:34]                   │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.siem-signals-default]
[00:10:34]                 │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.siem-signals-default]
[00:10:39]                 │ proc [kibana] [2021-10-15T09:02:42.989+00:00][INFO ][plugins.eventLog] event logged: {"@timestamp":"2021-10-15T09:02:42.988Z","event":{"provider":"alerting","action":"execute-start","kind":"alert","category":["siem"],"start":"2021-10-15T09:02:42.988Z"},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"a545c480-2d96-11ec-b69a-6ddd1fa9d788","type_id":"siem.signals"}],"task":{"scheduled":"2021-10-15T09:02:39.766Z","schedule_delay":3222000000},"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d","version":"8.0.0"},"rule":{"id":"a545c480-2d96-11ec-b69a-6ddd1fa9d788","license":"basic","category":"siem.signals","ruleset":"siem"},"message":"alert execution start: \"a545c480-2d96-11ec-b69a-6ddd1fa9d788\"","ecs":{"version":"1.8.0"}}
[00:10:42]                 │ proc [kibana] [2021-10-15T09:02:45.823+00:00][INFO ][plugins.securitySolution] [+] Finished indexing 88  signals searched between date ranges [
[00:10:42]                 │ proc [kibana]   {
[00:10:42]                 │ proc [kibana]     "to": "2021-10-15T09:02:44.815Z",
[00:10:42]                 │ proc [kibana]     "from": "1900-01-01T00:00:00.000Z",
[00:10:42]                 │ proc [kibana]     "maxSignals": 100
[00:10:42]                 │ proc [kibana]   }
[00:10:42]                 │ proc [kibana] ] name: "Query with a rule id" id: "a545c480-2d96-11ec-b69a-6ddd1fa9d788" rule id: "rule-1" signals index: ".siem-signals-default"
[00:10:42]                 │ proc [kibana] [2021-10-15T09:02:45.831+00:00][INFO ][plugins.eventLog] event logged: {"@timestamp":"2021-10-15T09:02:42.988Z","event":{"provider":"alerting","action":"execute","kind":"alert","category":["siem"],"start":"2021-10-15T09:02:42.988Z","outcome":"success","end":"2021-10-15T09:02:45.830Z","duration":2842000000},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"a545c480-2d96-11ec-b69a-6ddd1fa9d788","type_id":"siem.signals"}],"task":{"scheduled":"2021-10-15T09:02:39.766Z","schedule_delay":3222000000},"alerting":{"status":"ok"},"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d","version":"8.0.0"},"rule":{"id":"a545c480-2d96-11ec-b69a-6ddd1fa9d788","license":"basic","category":"siem.signals","ruleset":"siem","name":"Query with a rule id"},"message":"alert executed: siem.signals:a545c480-2d96-11ec-b69a-6ddd1fa9d788: 'Query with a rule id'","ecs":{"version":"1.8.0"}}
[00:10:42]                 └- ✖ fail: detection engine api security and spaces enabled  create_threat_matching tests with auditbeat data should be able to execute and get 10 signals when doing a specific query
[00:10:42]                 │      Error: expected undefined to be truthy
[00:10:42]                 │       at Assertion.assert (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:10:42]                 │       at Assertion.ok (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:122:8)
[00:10:42]                 │       at Function.ok (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:531:15)
[00:10:42]                 │       at Context.<anonymous> (test/detection_engine_api_integration/security_and_spaces/tests/create_threat_matching.ts:172:43)
[00:10:42]                 │       at runMicrotasks (<anonymous>)
[00:10:42]                 │       at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:10:42]                 │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:10:42]                 │ 
[00:10:42]                 │ 

Stack Trace

Error: expected undefined to be truthy
    at Assertion.assert (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.ok (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:122:8)
    at Function.ok (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/expect/expect.js:531:15)
    at Context.<anonymous> (test/detection_engine_api_integration/security_and_spaces/tests/create_threat_matching.ts:172:43)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@joshdover
Copy link
Contributor Author

Unrelated flaky test, merging.

@joshdover joshdover added the auto-backport Deprecated - use backport:version if exact versions are needed label Oct 15, 2021
@joshdover joshdover merged commit 033dfb3 into elastic:master Oct 15, 2021
@joshdover joshdover deleted the fix-111755 branch October 15, 2021 10:30
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Oct 15, 2021
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Oct 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.16.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Fleet] Switch to kibana_system user for setup & all package operations
5 participants