Skip to content

Commit

Permalink
[SIEM] Final Shimming/Prep for Server NP Migration (elastic#56814)
Browse files Browse the repository at this point in the history
* Route all our server setup through the plugin

Next we'll be trimming down ServerFacade to the final few dependencies,
and using our plugin dependencies for the rest.

* Clean up server plugin exports

For now, let's try to simplify our typings by exporting our plugin's
dependencies from the plugin itself, since we know it already knows
about them.

* Move DE Routes to to conventional location

I'm throwing the alerting registration in the plugin for now, too.

* Loosen up our RequestFacade

Now that we've audited our use of request objects, we can switch to the
more friendly LegacyRequest that all our utilities are expecting.
LegacyRequest doesn't have plugins, either, so our only remaining errant
usage is retrieving clients from it, which we call out explicitly.

* Remove uses of 'src' alias

This just threw an error on startup, so I'm fixing all of them to be
safe.

* Fix types of our GraphQL requests

These come through as new KibanaRequests and not as the LegacyRequest
that I had incorrectly typed them as.

I'm only caring about the `body` property right now, since that's all we
really deal with, and in testing it was all that was populated in our
actual requests, too.

* Initialize our routes with SetupServices

We're using the legacy version for now, but ServerFacade will be gone by
the end of the migration.

* Swap legacy spaces plugin for NP

This changes the signature of getIndex, which is used in quite a few
places. We'll see how bad that looks in the next commit.

* Remove unneeded typing

We're already ignoring another portion of the platform shim due to it
being incompatibly typed, so we might as well remove this.

* WIP: Converting our DE routes to use consolidated services

This contains our legacy stuff for now. Eventually, it will only be our
NP services. This breaks a few tests due to the way createMockServer
works, but I'll clean that up momentarily.

* Fix DE routing tests following refactor

The createMockServer helper does a few things differently:

* returns mocked LegacyServices that can be passed to our route
factories
* does not return a server object as we don't need it, except for:
* returns an inject function that we can use to execute a request

We're casting our services because we're only mocking a subset of what
LegacySetupServices entails.

Mainly, this allows me to continue moving
things off of ServerFacade without significant refactoring of tests.

* Fix incompatible request types

Unfortunately, LegacyRequest does not allow a request's `query` values
to be anything other than a string or string[]. However, in practice
they can (and are) objects, booleans, etc.

Our request types (e.g. QueryRequest) are correct, but because service
functions are LegacyRequest's implementation of `query`, we need to cast
them lest a type error occur.

For now, the easiest solution is to do this in the request handler:
intersecting with our RequestFacade (LegacyRequest) allows us to both a)
pluck our query params off the request and b) pass the request to NP
services.

* Move our use of encryptedSavedObjects to NP

We're just retrieving a boolean from it right now, which is also guarded
against the plugin being unavailable. If this usage becomes more
widespread, we'll make this available at a higher level, probably in
redux.

* Use NP elasticsearch client

* Simplifies our generic type to accept two arguments: params and the
return value
  * Options is fixed and we were never specifying anything meaningful
  there
* Updates all DE cluster calls to use callWithRequestFactory

* Update DE mocks with NP elasticsearch

* createMockServer now returns the callCluster mock, so that you can easily
mock a client response without needing to know the details of how we
define that function

* Remove savedObjects dependency from our legacy dependencies

This was added during a refactor, but we were never actually using this
code. We always retrieve the client from the request via
getSavedObjectsClient.

I think that the NP client has a slightly different interface, so we're
going to create a helper to retrieve it from the request, same as we do
with the elastic client.

In the future, both of these will be available on the request context,
and we can remove the helpers entirely.

* WIP: Convert services to stateful object

In trying to migrate over the savedObjectsClient, I realized that it is
not available during setup in the same way that the ES client is.

Since our routes need pieces from both setup and start phases, I've
added a Services class to accumulate/transform these services and expose
scoped clients when given a legacy request.

In New Platform, these clients will be available upon the request
context and we should be able to remove getScopedServicesFactory for our
routes. A subset of Services' functionality may still be useful, we'll
see.

* WIP: Converting routes to use Services factory

I decided that config shouldn't live in here, as this is only
client-related stuff. Probably going to rename this ClientsService.

Things are still very much broken.

* WIP: Qualifying our Service to ClientsService

This gets us client-related services (ES, SavedObjects, Alerts), but it
is independent of any configuration, which is gonna be another service.

* Fix types on getIndex function

This is a weird helper, I'm not really sure where it should go.

* Our ClientsService is a clients ... service

Return clients, as this is closer to what we'll get in the request
context.

* Clean up our server types

* Declare legacy types at top-level file
* Don't re-export from the plugin solely for convenience, that's a
slippery slope straight to circular dependencies
* Remove RequestFacade as it was a facade for LegacyRequest
* Rename ServerFacade/LegacySetupServices to just LegacyServices

* Refactor mocks for new architecture

* Separates config, server, and client mocks, as they're now independent
in our system, route-wise.
* gets one test working, the rest will follow.

* Simplify our routing mocks

* Adds mock for our new clients service
* Greatly simplifies both server and mock configs
* Renames factory method of client service

* Loosen graphQL endpoint validations

These work fine in production, but it's graphQL so we don't really need
the additional validation of these endpoints, and we weren't leveraging
these types anywhere in Typescript land.

Additionally, these restrictive validations prevent the initial
introspection calls done by graphiQL to get schema information, and
without schemae graphiql wasn't very helpful. This is a dev-only
problem, but that's the audience of graphiql.

* Remove unused graphql endpoint

This was only registered in dev mode; I thought that it was needed by
graphiql. However, after digging further I realized that graphiQL also
only makes POST calls to our real graphQL endpoint, so this route is
unnecessary.

* Reduce our dependence on PluginInitializerContext

After a little more introspection I realized our FrameworkAdapter
doesn't need the kibana version. It was only used in order to make a dev
request via (graphiql), but even that can be performed with a simpler
xsrf header.

This meant that we really only wanted to know whether we're in
production or not, so instead we pass that simple boolean to the
constructor.

* Fix FrameworkAdapter type

We no longer need this property.

* Update detections route tests

Uses the new routes interfaces, and our corresponding new mocks.

* Remove unnecessary null checks

Our savedObjectsClient is always going to be there.

* Remove unused type

YAGNI

* Remove unused savedObjects client

Turns out we were only destructuring this client for the null check.

* Handle case where spaces is disabled

We already null-coalesce properly in the clients service, but this
property access was missed.

* Return default signals index if spaces are disabled

* Remove unnecessary casting of our alerts client mock

I think that this was the result of us importing the wrong AlertsClient
type, or perhaps the types were out of sync. Regardless, they work now.

* Return the 'default' space even when spaces are disabled

This will allow users with spaces disabled to enable spaces without
losing data. The tradeoff is that they may be surprised when signals
don't exist within their configured xpack.siem.signalsIndex.

* Account for spaces being disabled in ClientsService

* Updates types to reflect that spaces may be unavailable
* Adds a test for getSpaceId's behavior when spaces are disabled

* Fix false positives in query signals routes tests

* Refactors mock expectations so that they're actually evaluated; they
can't go within a mockImplementation call as it's evaluated in the wrong
scope.
* Fixes duplicated test to use the proper assertions

* style: Prefer null coalescing over ternary
  • Loading branch information
rylnd authored Feb 14, 2020
1 parent b220454 commit 8513498
Show file tree
Hide file tree
Showing 93 changed files with 1,407 additions and 1,391 deletions.
34 changes: 12 additions & 22 deletions x-pack/legacy/plugins/siem/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,10 @@
*/

import { i18n } from '@kbn/i18n';
import { get } from 'lodash/fp';
import { resolve } from 'path';
import { Server } from 'hapi';
import { Root } from 'joi';

import { PluginInitializerContext } from '../../../../src/core/server';
import { plugin } from './server';
import { savedObjectMappings } from './server/saved_objects';

Expand All @@ -32,7 +30,6 @@ import {
SIGNALS_INDEX_KEY,
} from './common/constants';
import { defaultIndexPattern } from './default_index_pattern';
import { initServerWithKibana } from './server/kibana.index';
import { DEFAULT_APP_CATEGORIES } from '../../../../src/core/utils';

// eslint-disable-next-line @typescript-eslint/no-explicit-any
Expand Down Expand Up @@ -151,27 +148,20 @@ export const siem = (kibana: any) => {
mappings: savedObjectMappings,
},
init(server: Server) {
const { config, newPlatform, plugins, route } = server;
const { coreContext, env, setup } = newPlatform;
const initializerContext = { ...coreContext, env } as PluginInitializerContext;
const serverFacade = {
config,
usingEphemeralEncryptionKey:
get('usingEphemeralEncryptionKey', newPlatform.setup.plugins.encryptedSavedObjects) ??
false,
plugins: {
alerting: plugins.alerting,
actions: newPlatform.start.plugins.actions,
elasticsearch: plugins.elasticsearch,
spaces: plugins.spaces,
savedObjects: server.savedObjects.SavedObjectsClient,
},
route: route.bind(server),
const { coreContext, env, setup, start } = server.newPlatform;
const initializerContext = { ...coreContext, env };
const __legacy = {
config: server.config,
alerting: server.plugins.alerting,
route: server.route.bind(server),
};
// @ts-ignore-next-line: setup.plugins is too loosely typed
plugin(initializerContext).setup(setup.core, setup.plugins);

initServerWithKibana(initializerContext, serverFacade);
// @ts-ignore-next-line: NewPlatform shim is too loosely typed
const pluginInstance = plugin(initializerContext);
// @ts-ignore-next-line: NewPlatform shim is too loosely typed
pluginInstance.setup(setup.core, setup.plugins, __legacy);
// @ts-ignore-next-line: NewPlatform shim is too loosely typed
pluginInstance.start(start.core, start.plugins);
},
config(Joi: Root) {
// See x-pack/plugins/siem/server/config.ts if you're adding another
Expand Down
2 changes: 1 addition & 1 deletion x-pack/legacy/plugins/siem/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { PluginInitializerContext } from 'src/core/server';
import { PluginInitializerContext } from '../../../../../src/core/server';
import { Plugin } from './plugin';

export const plugin = (context: PluginInitializerContext) => {
Expand Down
86 changes: 0 additions & 86 deletions x-pack/legacy/plugins/siem/server/kibana.index.ts

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ describe('alerts elasticsearch_adapter', () => {
return mockAlertsHistogramDataResponse;
});
const mockFramework: FrameworkAdapter = {
version: 'mock',
callWithRequest: mockCallWithRequest,
registerGraphQLEndpoint: jest.fn(),
getIndexPatternsService: jest.fn(),
Expand Down
9 changes: 4 additions & 5 deletions x-pack/legacy/plugins/siem/server/lib/compose/kibana.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { CoreSetup, PluginInitializerContext } from '../../../../../../../src/core/server';
import { PluginsSetup } from '../../plugin';
import { CoreSetup, SetupPlugins } from '../../plugin';

import { Anomalies } from '../anomalies';
import { ElasticsearchAnomaliesAdapter } from '../anomalies/elasticsearch_adapter';
Expand Down Expand Up @@ -37,10 +36,10 @@ import { Alerts, ElasticsearchAlertsAdapter } from '../alerts';

export function compose(
core: CoreSetup,
plugins: PluginsSetup,
env: PluginInitializerContext['env']
plugins: SetupPlugins,
isProductionMode: boolean
): AppBackendLibs {
const framework = new KibanaBackendFrameworkAdapter(core, plugins, env);
const framework = new KibanaBackendFrameworkAdapter(core, plugins, isProductionMode);
const sources = new Sources(new ConfigurationSourcesAdapter());
const sourceStatus = new SourceStatus(new ElasticsearchSourceStatusAdapter(framework));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,13 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

// See the reference(s) below on explanations about why -000001 was chosen and
// why the is_write_index is true as well as the bootstrapping step which is needed.
// Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/applying-policy-to-template.html
export const createBootstrapIndex = async (
callWithRequest: CallWithRequest<
{ path: string; method: 'PUT'; body: unknown },
CallClusterOptions,
boolean
>,
callWithRequest: CallWithRequest<{ path: string; method: 'PUT'; body: unknown }, boolean>,
index: string
): Promise<unknown> => {
return callWithRequest('transport.request', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@
*/

import { IndicesDeleteParams } from 'elasticsearch';
import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

export const deleteAllIndex = async (
callWithRequest: CallWithRequest<IndicesDeleteParams, CallClusterOptions, boolean>,
callWithRequest: CallWithRequest<IndicesDeleteParams, boolean>,
index: string
): Promise<boolean> => {
return callWithRequest('indices.delete', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import { CallWithRequest } from '../types';

export const deletePolicy = async (
callWithRequest: CallWithRequest<{ path: string; method: 'DELETE' }, {}, unknown>,
callWithRequest: CallWithRequest<{ path: string; method: 'DELETE' }, unknown>,
policy: string
): Promise<unknown> => {
return callWithRequest('transport.request', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@
*/

import { IndicesDeleteTemplateParams } from 'elasticsearch';
import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

export const deleteTemplate = async (
callWithRequest: CallWithRequest<IndicesDeleteTemplateParams, CallClusterOptions, unknown>,
callWithRequest: CallWithRequest<IndicesDeleteTemplateParams, unknown>,
name: string
): Promise<unknown> => {
return callWithRequest('indices.deleteTemplate', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ import { CallWithRequest } from '../types';
export const getIndexExists = async (
callWithRequest: CallWithRequest<
{ index: string; size: number; terminate_after: number; allow_no_indices: boolean },
{},
{ _shards: { total: number } }
>,
index: string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import { CallWithRequest } from '../types';

export const getPolicyExists = async (
callWithRequest: CallWithRequest<{ path: string; method: 'GET' }, {}, unknown>,
callWithRequest: CallWithRequest<{ path: string; method: 'GET' }, unknown>,
policy: string
): Promise<boolean> => {
try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@
*/

import { IndicesExistsTemplateParams } from 'elasticsearch';
import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

export const getTemplateExists = async (
callWithRequest: CallWithRequest<IndicesExistsTemplateParams, CallClusterOptions, boolean>,
callWithRequest: CallWithRequest<IndicesExistsTemplateParams, boolean>,
template: string
): Promise<boolean> => {
return callWithRequest('indices.existsTemplate', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@
*/

import { IndicesGetSettingsParams } from 'elasticsearch';
import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

export const readIndex = async (
callWithRequest: CallWithRequest<IndicesGetSettingsParams, CallClusterOptions, unknown>,
callWithRequest: CallWithRequest<IndicesGetSettingsParams, unknown>,
index: string
): Promise<unknown> => {
return callWithRequest('indices.get', {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import { CallWithRequest } from '../types';

export const setPolicy = async (
callWithRequest: CallWithRequest<{ path: string; method: 'PUT'; body: unknown }, {}, unknown>,
callWithRequest: CallWithRequest<{ path: string; method: 'PUT'; body: unknown }, unknown>,
policy: string,
body: unknown
): Promise<unknown> => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@
*/

import { IndicesPutTemplateParams } from 'elasticsearch';
import { CallClusterOptions } from 'src/legacy/core_plugins/elasticsearch';
import { CallWithRequest } from '../types';

export const setTemplate = async (
callWithRequest: CallWithRequest<IndicesPutTemplateParams, CallClusterOptions, unknown>,
callWithRequest: CallWithRequest<IndicesPutTemplateParams, unknown>,
name: string,
body: unknown
): Promise<unknown> => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import { CallWithRequest } from '../types';

export const readPrivileges = async (
callWithRequest: CallWithRequest<unknown, unknown, unknown>,
callWithRequest: CallWithRequest<{}, unknown>,
index: string
): Promise<unknown> => {
return callWithRequest('transport.request', {
Expand Down
Loading

0 comments on commit 8513498

Please sign in to comment.