Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM] Final Shimming/Prep for Server NP Migration #56814

Merged
merged 43 commits into from
Feb 14, 2020

Conversation

rylnd
Copy link
Contributor

@rylnd rylnd commented Feb 5, 2020

Summary

This PR accomplishes the following goals:

  • Registers DE routes more similar to NP routes
    • Uses a thunk to split route registration between setup and start phases
      • We need the start phase right now in order to access the necessary clients (which are normalized under getClients). In NP, these clients will be available in the request handler context, and so the thunk (and likely getClients) can be removed.
    • Cleans up DE route tests accordingly
  • Fixes typing issues with Legacy requests
  • Migrates available services
    • spaces
    • elasticsearch
    • savedObjects
    • encryptedSavedObjects
    • config is staying for now, it'll be replaced with a simple object when we can leverage createConfig$

#45831

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

For maintainers

@rylnd rylnd added Team:SIEM v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.7.0 labels Feb 5, 2020
@rylnd rylnd self-assigned this Feb 5, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@rylnd rylnd force-pushed the final_backend_prep branch 2 times, most recently from 9f6bb66 to 945410b Compare February 8, 2020 20:47
rylnd added 21 commits February 10, 2020 12:17
Next we'll be trimming down ServerFacade to the final few dependencies,
and using our plugin dependencies for the rest.
For now, let's try to simplify our typings by exporting our plugin's
dependencies from the plugin itself, since we know it already knows
about them.
I'm throwing the alerting registration in the plugin for now, too.
Now that we've audited our use of request objects, we can switch to the
more friendly LegacyRequest that all our utilities are expecting.
LegacyRequest doesn't have plugins, either, so our only remaining errant
usage is retrieving clients from it, which we call out explicitly.
This just threw an error on startup, so I'm fixing all of them to be
safe.
These come through as new KibanaRequests and not as the LegacyRequest
that I had incorrectly typed them as.

I'm only caring about the `body` property right now, since that's all we
really deal with, and in testing it was all that was populated in our
actual requests, too.
We're using the legacy version for now, but ServerFacade will be gone by
the end of the migration.
This changes the signature of getIndex, which is used in quite a few
places. We'll see how bad that looks in the next commit.
We're already ignoring another portion of the platform shim due to it
being incompatibly typed, so we might as well remove this.
This contains our legacy stuff for now. Eventually, it will only be our
NP services. This breaks a few tests due to the way createMockServer
works, but I'll clean that up momentarily.
The createMockServer helper does a few things differently:

* returns mocked LegacyServices that can be passed to our route
factories
* does not return a server object as we don't need it, except for:
* returns an inject function that we can use to execute a request

We're casting our services because we're only mocking a subset of what
LegacySetupServices entails.

Mainly, this allows me to continue moving
things off of ServerFacade without significant refactoring of tests.
Unfortunately, LegacyRequest does not allow a request's `query` values
to be anything other than a string or string[]. However, in practice
they can (and are) objects, booleans, etc.

Our request types (e.g. QueryRequest) are correct, but because service
functions are LegacyRequest's implementation of `query`, we need to cast
them lest a type error occur.

For now, the easiest solution is to do this in the request handler:
intersecting with our RequestFacade (LegacyRequest) allows us to both a)
pluck our query params off the request and b) pass the request to NP
services.
We're just retrieving a boolean from it right now, which is also guarded
against the plugin being unavailable. If this usage becomes more
widespread, we'll make this available at a higher level, probably in
redux.
* Simplifies our generic type to accept two arguments: params and the
return value
  * Options is fixed and we were never specifying anything meaningful
  there
* Updates all DE cluster calls to use callWithRequestFactory
* createMockServer now returns the callCluster mock, so that you can easily
mock a client response without needing to know the details of how we
define that function
This was added during a refactor, but we were never actually using this
code. We always retrieve the client from the request via
getSavedObjectsClient.

I think that the NP client has a slightly different interface, so we're
going to create a helper to retrieve it from the request, same as we do
with the elastic client.

In the future, both of these will be available on the request context,
and we can remove the helpers entirely.
In trying to migrate over the savedObjectsClient, I realized that it is
not available during setup in the same way that the ES client is.

Since our routes need pieces from both setup and start phases, I've
added a Services class to accumulate/transform these services and expose
scoped clients when given a legacy request.

In New Platform, these clients will be available upon the request
context and we should be able to remove getScopedServicesFactory for our
routes. A subset of Services' functionality may still be useful, we'll
see.
I decided that config shouldn't live in here, as this is only
client-related stuff. Probably going to rename this ClientsService.

Things are still very much broken.
This gets us client-related services (ES, SavedObjects, Alerts), but it
is independent of any configuration, which is gonna be another service.
This is a weird helper, I'm not really sure where it should go.
Return clients, as this is closer to what we'll get in the request
context.
@rylnd rylnd marked this pull request as ready for review February 12, 2020 21:08
 Conflicts:
	x-pack/legacy/plugins/siem/server/kibana.index.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/create_index_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/delete_index_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/add_prepackaged_rules_route.test.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/add_prepackaged_rules_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/create_rules_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/delete_rules_bulk_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/find_rules_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_bulk_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/types.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/types.ts
 Conflicts:
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/utils.test.ts
	x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/utils.ts
* Updates types to reflect that spaces may be unavailable
* Adds a test for getSpaceId's behavior when spaces are disabled
@rylnd rylnd force-pushed the final_backend_prep branch from 99d5a78 to 668702b Compare February 13, 2020 14:40
@@ -4,6 +4,7 @@
* you may not use this file except in compliance with the Elastic License.
*/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious - should this import use 'kibana/server' instead of the relative import? I remember seeing some sort of message saying when to use one over the other but not sure what situation warrants what.

const pluginInstance = plugin(initializerContext);
// @ts-ignore-next-line: NewPlatform shim is too loosely typed
pluginInstance.setup(setup.core, setup.plugins, __legacy);
// @ts-ignore-next-line: NewPlatform shim is too loosely typed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ouch on having to do ts-ignore-next-line. If we have to, we have to. Just some ouch. Any open tickets or other information about when and how these can be removed or is this is just temporary for a shim that will be altogether removed once we are out of the legacy folder?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is specific to the shim; this entire file will be removed on new platform.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few of these (CoreStart, CoreSetup) are properly typed, but because of the dynamic DI the plugins types are understandably loose. We could make this better, but we'd still have to cast parts of it, and this is all temporary code.

const callWithRequest = callWithRequestFactory(request, server);
const indexExists = await getIndexExists(callWithRequest, finalIndex);
const finalIndex =
outputIndex != null ? outputIndex : getIndex(spacesClient.getSpaceId, config);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: I think we can use the fancy smancy ?? here? Otherwise still looks good, sorry you're refactoring my code. I have old habits I am trying to stop doing such as this one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FrankHassanabad In the course of making these changes I think I might've found a bug: https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/rules/import_rules_route.ts#L149 does not seem to respect an imported rule's output index.


export const createReadTagsRoute: Hapi.ServerRoute = {
export const createReadTagsRoute = (getClients: GetScopedClients): Hapi.ServerRoute => ({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just throwing this out there -> we are mixing ({}) and { return {} } in between the routes. Not sure if we want to take this opportunity to create some consistency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I prefer the former unless there's a good reason. In the interest of this PR, the reason was not introducing a lot of whitespace changes: if converting caused everything to be re-indented, I didn't do it.

Copy link
Contributor

@FrankHassanabad FrankHassanabad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments but feel free with how big this is that maybe some of these issues pre-date this ticket so feel free to create new bugs from what you find and merge this now or fix the few things found and then merge. Up to you, but the sooner this gets in the better chance of not having another conflict you will have.

Overall, really like how large swaths of code were removed in the mocks and how this shim gets us closer to our goals of moving out of the legacy folder.

LGTM, thanks for all the disciple needed to keep all this stuff straight while we were merging code while this was still in progress.

Copy link
Contributor

@dhurley14 dhurley14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left another comment but this looks great as-is! Thanks for juggling the changes during this process while we were developing detection engine.

* Refactors mock expectations so that they're actually evaluated; they
can't go within a mockImplementation call as it's evaluated in the wrong
scope.
* Fixes duplicated test to use the proper assertions
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/dashboard/feature_controls/dashboard_spaces·ts.dashboard feature controls spaces space with no features disabled landing page shows "Create new Dashboard" button

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: dashboard
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: feature controls
[00:00:00]             └-> "before all" hook
[00:06:49]             └-: spaces
[00:06:49]               └-> "before all" hook
[00:06:49]               └-> "before all" hook
[00:06:49]                 │ info [logstash_functional] Loading "mappings.json"
[00:06:49]                 │ info [logstash_functional] Loading "data.json.gz"
[00:06:49]                 │ info [logstash_functional] Skipped restore for existing index "logstash-2015.09.22"
[00:06:49]                 │ info [logstash_functional] Skipped restore for existing index "logstash-2015.09.20"
[00:06:49]                 │ info [logstash_functional] Skipped restore for existing index "logstash-2015.09.21"
[00:06:51]               └-: space with no features disabled
[00:06:51]                 └-> "before all" hook
[00:06:51]                 └-> "before all" hook
[00:06:51]                   │ info [dashboard/feature_controls/spaces] Loading "mappings.json"
[00:06:51]                   │ info [dashboard/feature_controls/spaces] Loading "data.json"
[00:06:51]                   │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/jz-uzjyHT2eA0qhg4ZP5Kw] deleting index
[00:06:51]                   │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_1/GLG3CbItSdOtu5WJCM-nGw] deleting index
[00:06:51]                   │ info [dashboard/feature_controls/spaces] Deleted existing index [".kibana_2",".kibana_1"]
[00:06:51]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:06:52]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana][0]]]).
[00:06:52]                   │ info [dashboard/feature_controls/spaces] Created index ".kibana"
[00:06:52]                   │ debg [dashboard/feature_controls/spaces] ".kibana" settings {"index":{"number_of_shards":"1","auto_expand_replicas":"0-1","number_of_replicas":"0"}}
[00:06:52]                   │ info [dashboard/feature_controls/spaces] Indexed 6 docs into ".kibana"
[00:06:53]                   │ info Creating index .kibana_2.
[00:06:53]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:06:53]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] updating number_of_replicas to [0] for indices [.kibana_2]
[00:06:53]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_2][0]]]).
[00:06:53]                   │ info Reindexing .kibana to .kibana_1
[00:06:53]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:06:53]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] updating number_of_replicas to [0] for indices [.kibana_1]
[00:06:53]                   │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]]]).
[00:06:53]                   │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] 7497 finished with response BulkByScrollResponse[took=69.6ms,timed_out=false,sliceId=null,updated=0,created=6,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:06:53]                   │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana/TDpjMT7kSGSDeLXIAPAR0g] deleting index
[00:06:53]                   │ info Migrating .kibana_1 saved objects to .kibana_2
[00:06:53]                   │ debg Migrating saved objects index-pattern:logstash-*, config:6.0.0, custom_space:index-pattern:logstash-*, custom_space:visualization:75c3e060-1e7c-11e9-8488-65449e65d0ed, custom_space:dashboard:i-exist, custom_space:config:6.0.0
[00:06:54]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:54]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:54]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:54]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:54]                   │ info Pointing alias .kibana to .kibana_2.
[00:06:54]                   │ info Finished in 1041ms.
[00:06:54]                   │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:06:55]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:56]                   │ debg creating space
[00:06:56]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:06:57]                   │ debg created space
[00:06:57]                 └-> shows dashboard navlink
[00:06:57]                   └-> "before each" hook: global before each
[00:06:57]                   │ debg navigating to home url: http://localhost:6141/s/custom_space/app/kibana#/home
[00:06:57]                   │ debg Navigate to: http://localhost:6141/s/custom_space/app/kibana#/home
[00:06:57]                   │ debg ... sleep(700) start
[00:06:57]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/login?next=%2Fs%2Fcustom_space%2Fapp%2Fkibana%3F_t%3D1581639681028#/home 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:06:57]                   │
[00:06:57]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/app/login/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:06:58]                   │ debg ... sleep(700) end
[00:06:58]                   │ debg returned from get, calling refresh
[00:06:58]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/login?next=%2Fs%2Fcustom_space%2Fapp%2Fkibana%3F_t%3D1581639681028#/home 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:06:58]                   │
[00:06:58]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/app/login/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:06:58]                   │ debg currentUrl = http://localhost:6141/s/custom_space/login?next=%2Fs%2Fcustom_space%2Fapp%2Fkibana%3F_t%3D1581639681028#/home
[00:06:58]                   │          appUrl = http://localhost:6141/s/custom_space/app/kibana#/home
[00:06:58]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:07:06]                   │ debg Found login page.  Logging in with username = elastic
[00:07:06]                   │ debg TestSubjects.setValue(loginUsername, elastic)
[00:07:06]                   │ debg TestSubjects.click(loginUsername)
[00:07:06]                   │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:07:06]                   │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:07:06]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-14T00:21:29Z
[00:07:06]                   │        Adding connection to http://localhost:6141/s/custom_space/elasticsearch
[00:07:06]                   │
[00:07:06]                   │      "
[00:07:06]                   │ debg TestSubjects.setValue(loginPassword, changeme)
[00:07:06]                   │ debg TestSubjects.click(loginPassword)
[00:07:06]                   │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:07:06]                   │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:07:07]                   │ debg TestSubjects.click(loginSubmit)
[00:07:07]                   │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:07:07]                   │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:07:07]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] nav:not(.ng-hide)') with timeout=20000
[00:07:18]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/app/kibana?_t=1581639681028#/home 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:07:18]                   │
[00:07:18]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:07:18]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-14T00:21:39Z
[00:07:18]                   │        Adding connection to http://localhost:6141/s/custom_space/elasticsearch
[00:07:18]                   │
[00:07:18]                   │      "
[00:07:18]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/app/kibana?_t=1581639701894#/home 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:07:18]                   │
[00:07:18]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:07:18]                   │ debg Finished login process currentUrl = http://localhost:6141/s/custom_space/app/kibana#/home
[00:07:18]                   │ debg TestSubjects.find(kibanaChrome)
[00:07:18]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=10000
[00:07:28]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-14T00:21:50Z
[00:07:28]                   │        Adding connection to http://localhost:6141/s/custom_space/elasticsearch
[00:07:28]                   │
[00:07:28]                   │      "
[00:07:28]                   │ debg ... sleep(501) start
[00:07:29]                   │ debg ... sleep(501) end
[00:07:29]                   │ debg in navigateTo url = http://localhost:6141/s/custom_space/app/kibana#/home
[00:07:29]                   │ debg TestSubjects.exists(statusPageContainer)
[00:07:29]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:07:31]                   │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:07:32]                   │ debg navigating to settings url: http://localhost:6141/app/kibana#/management
[00:07:32]                   │ debg Navigate to: http://localhost:6141/app/kibana#/management
[00:07:33]                   │ debg ... sleep(700) start
[00:07:33]                   │ debg browser[INFO] http://localhost:6141/app/kibana?_t=1581639715590#/management 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:07:33]                   │
[00:07:33]                   │ debg browser[INFO] http://localhost:6141/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:07:33]                   │ debg ... sleep(700) end
[00:07:33]                   │ debg returned from get, calling refresh
[00:07:34]                   │ debg browser[INFO] http://localhost:6141/app/kibana?_t=1581639715590#/management 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:07:34]                   │
[00:07:34]                   │ debg browser[INFO] http://localhost:6141/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:07:35]                   │ debg currentUrl = http://localhost:6141/app/kibana#/management
[00:07:35]                   │          appUrl = http://localhost:6141/app/kibana#/management
[00:07:35]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:07:43]                   │ debg TestSubjects.find(kibanaChrome)
[00:07:43]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=10000
[00:07:43]                   │ debg browser[INFO] http://localhost:6141/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-14T00:22:05Z
[00:07:43]                   │        Adding connection to http://localhost:6141/elasticsearch
[00:07:43]                   │
[00:07:43]                   │      "
[00:07:43]                   │ debg ... sleep(501) start
[00:07:44]                   │ debg ... sleep(501) end
[00:07:44]                   │ debg in navigateTo url = http://localhost:6141/app/kibana#/management?_g=()
[00:07:44]                   │ debg TestSubjects.exists(statusPageContainer)
[00:07:44]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:07:46]                   │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:07:47]                   │ debg TestSubjects.click(settings)
[00:07:47]                   │ debg Find.clickByCssSelector('[data-test-subj="settings"]') with timeout=10000
[00:07:47]                   │ debg Find.findByCssSelector('[data-test-subj="settings"]') with timeout=10000
[00:07:49]                   │ debg isGlobalLoadingIndicatorVisible
[00:07:49]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:07:49]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:07:51]                   │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:07:51]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:07:51]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:07:51]                   │ debg TestSubjects.exists(managementSettingsTitle)
[00:07:51]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="managementSettingsTitle"]') with timeout=120000
[00:07:51]                   │ debg Find.clickByCssSelector('[data-test-subj="advancedSetting-editField-pageNavigation"] option[value="individual"]') with timeout=10000
[00:07:51]                   │ debg Find.findByCssSelector('[data-test-subj="advancedSetting-editField-pageNavigation"] option[value="individual"]') with timeout=10000
[00:07:52]                   │ debg isGlobalLoadingIndicatorVisible
[00:07:52]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:07:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:07:53]                   │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:07:54]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:07:54]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:07:54]                   │ debg TestSubjects.click(advancedSetting-saveEditField-pageNavigation)
[00:07:54]                   │ debg Find.clickByCssSelector('[data-test-subj="advancedSetting-saveEditField-pageNavigation"]') with timeout=10000
[00:07:54]                   │ debg Find.findByCssSelector('[data-test-subj="advancedSetting-saveEditField-pageNavigation"]') with timeout=10000
[00:07:54]                   │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1581637179657973492] [.kibana_2/8pIp7_kVSZWJS91eRr4Nwg] update_mapping [_doc]
[00:07:54]                   │ debg isGlobalLoadingIndicatorVisible
[00:07:54]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:07:54]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:07:54]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:07:54]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:07:54]                   │ debg TestSubjects.find(navDrawer)
[00:07:54]                   │ debg Find.findByCssSelector('[data-test-subj="navDrawer"]') with timeout=10000
[00:07:54]                   └- ✓ pass  (57.2s) "dashboard feature controls spaces space with no features disabled shows dashboard navlink"
[00:07:54]                 └-> landing page shows "Create new Dashboard" button
[00:07:54]                   └-> "before each" hook: global before each
[00:07:54]                   │ debg navigateToActualUrl http://localhost:6141/s/custom_space/app/kibana#/dashboards
[00:07:54]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/app/kibana?_t=1581639738259#/dashboards 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:07:54]                   │
[00:07:54]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:07:54]                   │ debg TestSubjects.exists(dashboardLandingPage)
[00:07:54]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=10000
[00:07:59]                   │ debg --- retry.tryForTime error: [data-test-subj="dashboardLandingPage"] is not displayed
[00:08:04]                   │ debg browser[INFO] http://localhost:6141/s/custom_space/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-14T00:22:26Z
[00:08:04]                   │        Adding connection to http://localhost:6141/s/custom_space/elasticsearch
[00:08:04]                   │
[00:08:04]                   │      "
[00:08:04]                   │ debg --- retry.tryForTime failed again with the same message...
[00:08:05]                   │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/dashboard feature controls spaces space with no features disabled landing page shows _Create new Dashboard_ button.png"
[00:08:05]                   │ info Current URL is: http://localhost:6141/s/custom_space/app/kibana#/dashboards?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:08:05]                   │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/dashboard feature controls spaces space with no features disabled landing page shows _Create new Dashboard_ button.html
[00:08:05]                   └- ✖ fail: "dashboard feature controls spaces space with no features disabled landing page shows "Create new Dashboard" button"
[00:08:05]                   │

Stack Trace

Error: expected testSubject(dashboardLandingPage) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/kibana/test/functional/services/test_subjects.ts:60:15)

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rylnd rylnd merged commit 8513498 into elastic:master Feb 14, 2020
@rylnd rylnd deleted the final_backend_prep branch February 14, 2020 04:10
yctercero added a commit to yctercero/kibana that referenced this pull request Feb 14, 2020
rylnd added a commit that referenced this pull request Feb 14, 2020
* Route all our server setup through the plugin

Next we'll be trimming down ServerFacade to the final few dependencies,
and using our plugin dependencies for the rest.

* Clean up server plugin exports

For now, let's try to simplify our typings by exporting our plugin's
dependencies from the plugin itself, since we know it already knows
about them.

* Move DE Routes to to conventional location

I'm throwing the alerting registration in the plugin for now, too.

* Loosen up our RequestFacade

Now that we've audited our use of request objects, we can switch to the
more friendly LegacyRequest that all our utilities are expecting.
LegacyRequest doesn't have plugins, either, so our only remaining errant
usage is retrieving clients from it, which we call out explicitly.

* Remove uses of 'src' alias

This just threw an error on startup, so I'm fixing all of them to be
safe.

* Fix types of our GraphQL requests

These come through as new KibanaRequests and not as the LegacyRequest
that I had incorrectly typed them as.

I'm only caring about the `body` property right now, since that's all we
really deal with, and in testing it was all that was populated in our
actual requests, too.

* Initialize our routes with SetupServices

We're using the legacy version for now, but ServerFacade will be gone by
the end of the migration.

* Swap legacy spaces plugin for NP

This changes the signature of getIndex, which is used in quite a few
places. We'll see how bad that looks in the next commit.

* Remove unneeded typing

We're already ignoring another portion of the platform shim due to it
being incompatibly typed, so we might as well remove this.

* WIP: Converting our DE routes to use consolidated services

This contains our legacy stuff for now. Eventually, it will only be our
NP services. This breaks a few tests due to the way createMockServer
works, but I'll clean that up momentarily.

* Fix DE routing tests following refactor

The createMockServer helper does a few things differently:

* returns mocked LegacyServices that can be passed to our route
factories
* does not return a server object as we don't need it, except for:
* returns an inject function that we can use to execute a request

We're casting our services because we're only mocking a subset of what
LegacySetupServices entails.

Mainly, this allows me to continue moving
things off of ServerFacade without significant refactoring of tests.

* Fix incompatible request types

Unfortunately, LegacyRequest does not allow a request's `query` values
to be anything other than a string or string[]. However, in practice
they can (and are) objects, booleans, etc.

Our request types (e.g. QueryRequest) are correct, but because service
functions are LegacyRequest's implementation of `query`, we need to cast
them lest a type error occur.

For now, the easiest solution is to do this in the request handler:
intersecting with our RequestFacade (LegacyRequest) allows us to both a)
pluck our query params off the request and b) pass the request to NP
services.

* Move our use of encryptedSavedObjects to NP

We're just retrieving a boolean from it right now, which is also guarded
against the plugin being unavailable. If this usage becomes more
widespread, we'll make this available at a higher level, probably in
redux.

* Use NP elasticsearch client

* Simplifies our generic type to accept two arguments: params and the
return value
  * Options is fixed and we were never specifying anything meaningful
  there
* Updates all DE cluster calls to use callWithRequestFactory

* Update DE mocks with NP elasticsearch

* createMockServer now returns the callCluster mock, so that you can easily
mock a client response without needing to know the details of how we
define that function

* Remove savedObjects dependency from our legacy dependencies

This was added during a refactor, but we were never actually using this
code. We always retrieve the client from the request via
getSavedObjectsClient.

I think that the NP client has a slightly different interface, so we're
going to create a helper to retrieve it from the request, same as we do
with the elastic client.

In the future, both of these will be available on the request context,
and we can remove the helpers entirely.

* WIP: Convert services to stateful object

In trying to migrate over the savedObjectsClient, I realized that it is
not available during setup in the same way that the ES client is.

Since our routes need pieces from both setup and start phases, I've
added a Services class to accumulate/transform these services and expose
scoped clients when given a legacy request.

In New Platform, these clients will be available upon the request
context and we should be able to remove getScopedServicesFactory for our
routes. A subset of Services' functionality may still be useful, we'll
see.

* WIP: Converting routes to use Services factory

I decided that config shouldn't live in here, as this is only
client-related stuff. Probably going to rename this ClientsService.

Things are still very much broken.

* WIP: Qualifying our Service to ClientsService

This gets us client-related services (ES, SavedObjects, Alerts), but it
is independent of any configuration, which is gonna be another service.

* Fix types on getIndex function

This is a weird helper, I'm not really sure where it should go.

* Our ClientsService is a clients ... service

Return clients, as this is closer to what we'll get in the request
context.

* Clean up our server types

* Declare legacy types at top-level file
* Don't re-export from the plugin solely for convenience, that's a
slippery slope straight to circular dependencies
* Remove RequestFacade as it was a facade for LegacyRequest
* Rename ServerFacade/LegacySetupServices to just LegacyServices

* Refactor mocks for new architecture

* Separates config, server, and client mocks, as they're now independent
in our system, route-wise.
* gets one test working, the rest will follow.

* Simplify our routing mocks

* Adds mock for our new clients service
* Greatly simplifies both server and mock configs
* Renames factory method of client service

* Loosen graphQL endpoint validations

These work fine in production, but it's graphQL so we don't really need
the additional validation of these endpoints, and we weren't leveraging
these types anywhere in Typescript land.

Additionally, these restrictive validations prevent the initial
introspection calls done by graphiQL to get schema information, and
without schemae graphiql wasn't very helpful. This is a dev-only
problem, but that's the audience of graphiql.

* Remove unused graphql endpoint

This was only registered in dev mode; I thought that it was needed by
graphiql. However, after digging further I realized that graphiQL also
only makes POST calls to our real graphQL endpoint, so this route is
unnecessary.

* Reduce our dependence on PluginInitializerContext

After a little more introspection I realized our FrameworkAdapter
doesn't need the kibana version. It was only used in order to make a dev
request via (graphiql), but even that can be performed with a simpler
xsrf header.

This meant that we really only wanted to know whether we're in
production or not, so instead we pass that simple boolean to the
constructor.

* Fix FrameworkAdapter type

We no longer need this property.

* Update detections route tests

Uses the new routes interfaces, and our corresponding new mocks.

* Remove unnecessary null checks

Our savedObjectsClient is always going to be there.

* Remove unused type

YAGNI

* Remove unused savedObjects client

Turns out we were only destructuring this client for the null check.

* Handle case where spaces is disabled

We already null-coalesce properly in the clients service, but this
property access was missed.

* Return default signals index if spaces are disabled

* Remove unnecessary casting of our alerts client mock

I think that this was the result of us importing the wrong AlertsClient
type, or perhaps the types were out of sync. Regardless, they work now.

* Return the 'default' space even when spaces are disabled

This will allow users with spaces disabled to enable spaces without
losing data. The tradeoff is that they may be surprised when signals
don't exist within their configured xpack.siem.signalsIndex.

* Account for spaces being disabled in ClientsService

* Updates types to reflect that spaces may be unavailable
* Adds a test for getSpaceId's behavior when spaces are disabled

* Fix false positives in query signals routes tests

* Refactors mock expectations so that they're actually evaluated; they
can't go within a mockImplementation call as it's evaluated in the wrong
scope.
* Fixes duplicated test to use the proper assertions

* style: Prefer null coalescing over ternary

Co-authored-by: Elastic Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:SIEM v7.7.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants