Merge branch 'master' into implement/kbn-client-import-export

docs/settings/alert-action-settings.asciidoc

@@ -21,9 +21,12 @@ You can configure the following settings in the `kibana.yml` file.
 [cols="2*<"]
 |===
 
-| `xpack.encryptedSavedObjects.encryptionKey`
+| `xpack.encryptedSavedObjects`
+`.encryptionKey`
   | A string of 32 or more characters used to encrypt sensitive properties on alerts and actions before they're stored in {es}. Third party credentials &mdash; such as the username and password used to connect to an SMTP service &mdash; are an example of encrypted properties. +
   +
+  {kib} offers a <<kibana-encryption-keys, CLI tool>> to help generate this encryption key. +
+  +
   If not set, {kib} will generate a random key on startup, but all alert and action functions will be blocked. Generated keys are not allowed for alerts and actions because when a new key is generated on restart, existing encrypted data becomes inaccessible. For the same reason, alerts and actions in high-availability deployments of {kib} will behave unexpectedly if the key isn't the same on all instances of {kib}. +
   +
   Although the key can be specified in clear text in `kibana.yml`, it's recommended to store this key securely in the <<secure-settings,{kib} Keystore>>.

docs/user/alerting/defining-alerts.asciidoc

@@ -59,7 +59,7 @@ image::images/alert-flyout-action-type-selection.png[UI for selecting an action
 When an alert instance matches a condition, the alert is marked as _Active_ and assigned an action group.  The actions in that group are triggered.
 When the condition is no longer detected, the alert is assigned to the _Recovered_ action group, which triggers any actions assigned to that group.
 
-**Run When** allows you to assign an action to an _action group_. This will trigger the action in accordance with your **Notify every** setting.
+**Run When** allows you to assign an action to an action group. This will trigger the action in accordance with your **Notify** setting.
 
 Each action must specify a <<alerting-concepts-connectors, connector>> instance. If no connectors exist for that action type, click *Add action* to create one.
 
@@ -68,7 +68,20 @@ Each action type exposes different properties. For example an email action allow
 [role="screenshot"]
 image::images/alert-flyout-action-details.png[UI for defining an email action]
 
-Using the https://mustache.github.io/[Mustache] template syntax `{{variable name}}`, you can pass alert values at the time a condition is detected to an action. Available variables differ by alert type, and the list of available variables can be accessed using the "add variable" button.
+[float]
+==== Action variables
+Using the https://mustache.github.io/[Mustache] template syntax `{{variable name}}`, you can pass alert values at the time a condition is detected to an action. You can access the list of available variables using the "add variable" button. Although available variables differ by alert type, all alert types pass the following variables:
+
+`alertId`:: The ID of the alert.
+`alertName`:: The name of the alert.
+`spaceId`:: The ID of the space for the alert.
+`tags`:: The list of tags applied to the alert.
+`date`:: The date the alert scheduled the action, in ISO format.
+`alertInstanceId`:: The ID of the alert instance that scheduled the action.
+`alertActionGroup`:: The ID of the action group of the alert instance that scheduled the action.
+`alertActionSubgroup`:: The action subgroup of the alert instance that scheduled the action.
+`alertActionGroupName`:: The name of the action group of the alert instance that scheduled the action.
+`kibanaBaseUrl`:: The configured <<server-publicBaseUrl, `server.publicBaseUrl`>>. If not configured, this will be empty.
 
 [role="screenshot"]
 image::images/alert-flyout-action-variables.png[Passing alert values to an action]

docs/user/alerting/stack-alerts/es-query.asciidoc

@@ -28,6 +28,27 @@ condition. Aggregations are not supported at this time.
 Threshold:: This clause defines a threshold value and a comparison operator  (`is above`, `is above or equals`, `is below`, `is below or equals`, or `is between`). The number of documents that match the specified query is compared to this threshold.
 Time window:: This clause determines how far back to search for documents, using the *time field* set in the *index* clause. Generally this value should be set to a value higher than the *check every* value in the <<defining-alerts-general-details, general alert details>>, to avoid gaps in detection. 
 
+[float]
+==== Action variables
+
+When the ES query alert condition is met, the following variables are available to use inside each action:
+
+`context.title`:: A preconstructed title for the alert. Example: `alert term match alert query matched`.
+`context.message`:: A preconstructed message for the alert. Example: +
+`alert 'term match alert' is active:` +
+`- Value: 42` +
+`- Conditions Met: count greater than 4 over 5m` +
+`- Timestamp: 2020-01-01T00:00:00.000Z`
+
+`context.group`:: The name of the action group associated with the condition. Example: `query matched`.
+`context.date`:: The date, in ISO format, that the alert met the condition. Example: `2020-01-01T00:00:00.000Z`.
+`context.value`:: The value of the alert that met the condition.
+`context.conditions`:: A description of the condition. Example: `count greater than 4`.
+`context.hits`:: The most recent ES documents that matched the query. Using the https://mustache.github.io/[Mustache] template array syntax, you can iterate over these hits to get values from the ES documents into your actions.
+
+[role="screenshot"]
+image::images/alert-types-es-query-example-action-variable.png[Iterate over hits using Mustache template syntax]
+
 [float]
 ==== Testing your query
 

docs/user/alerting/stack-alerts/index-threshold.asciidoc

@@ -31,6 +31,23 @@ If data is available and all clauses have been defined, a preview chart will ren
 [role="screenshot"]
 image::user/alerting/images/alert-types-index-threshold-preview.png[Five clauses define the condition to detect]
 
+[float]
+==== Action variables
+
+When the index threshold alert condition is met, the following variables are available to use inside each action:
+
+`context.title`:: A preconstructed title for the alert. Example: `alert kibana sites - high egress met threshold`.
+`context.message`:: A preconstructed message for the alert. Example: +
+`alert 'kibana sites - high egress' is active for group 'threshold met':` +
+`- Value: 42` +
+`- Conditions Met: count greater than 4 over 5m` +
+`- Timestamp: 2020-01-01T00:00:00.000Z`
+
+`context.group`:: The name of the action group associated with the threshold condition. Example: `threshold met`.
+`context.date`:: The date, in ISO format, that the alert met the threshold condition. Example: `2020-01-01T00:00:00.000Z`.
+`context.value`:: The value for the alert that met the threshold condition.
+`context.conditions`:: A description of the threshold condition. Example: `count greater than 4`
+
 [float]
 ==== Example
 

package.json

@@ -74,11 +74,11 @@
     "**/cross-fetch/node-fetch": "^2.6.1",
     "**/deepmerge": "^4.2.2",
     "**/fast-deep-equal": "^3.1.1",
-    "**/graphql-toolkit/lodash": "^4.17.15",
+    "**/graphql-toolkit/lodash": "^4.17.21",
     "**/hoist-non-react-statics": "^3.3.2",
     "**/isomorphic-fetch/node-fetch": "^2.6.1",
     "**/istanbul-instrumenter-loader/schema-utils": "1.0.0",
-    "**/load-grunt-config/lodash": "^4.17.20",
+    "**/load-grunt-config/lodash": "^4.17.21",
     "**/minimist": "^1.2.5",
     "**/node-jose/node-forge": "^0.10.0",
     "**/prismjs": "1.22.0",
@@ -233,7 +233,7 @@
     "json-stringify-safe": "5.0.1",
     "jsonwebtoken": "^8.5.1",
     "load-json-file": "^6.2.0",
-    "lodash": "^4.17.20",
+    "lodash": "^4.17.21",
     "lru-cache": "^4.1.5",
     "markdown-it": "^10.0.0",
     "md5": "^2.1.0",
@@ -390,7 +390,6 @@
     "@storybook/addon-essentials": "^6.0.26",
     "@storybook/addon-knobs": "^6.0.26",
     "@storybook/addon-storyshots": "^6.0.26",
-    "@storybook/addons": "^6.0.16",
     "@storybook/components": "^6.0.26",
     "@storybook/core": "^6.0.26",
     "@storybook/core-events": "^6.0.26",

src/dev/ci_setup/setup.sh

@@ -21,6 +21,10 @@ cp "src/dev/ci_setup/.bazelrc-ci" "$HOME/.bazelrc";
 echo "# Appended by src/dev/ci_setup/setup.sh" >> "$HOME/.bazelrc"
 echo "build --remote_header=x-buildbuddy-api-key=$KIBANA_BUILDBUDDY_CI_API_KEY" >> "$HOME/.bazelrc"
 
+if [[ "$BUILD_TS_REFS_CACHE_ENABLE" != "true" ]]; then
+  export BUILD_TS_REFS_CACHE_ENABLE=false
+fi
+
 ###
 ### install dependencies
 ###

src/dev/typescript/build_ts_refs_cli.ts

@@ -23,7 +23,7 @@ export async function runBuildRefsCli() {
     async ({ log, flags }) => {
       const outDirs = getOutputsDeep(REF_CONFIG_PATHS);
 
-      const cacheEnabled = process.env.BUILD_TS_REFS_CACHE_ENABLE === 'true' || !!flags.cache;
+      const cacheEnabled = process.env.BUILD_TS_REFS_CACHE_ENABLE !== 'false' && !!flags.cache;
       const doCapture = process.env.BUILD_TS_REFS_CACHE_CAPTURE === 'true';
       const doClean = !!flags.clean || doCapture;
       const doInitCache = cacheEnabled && !doClean;
@@ -62,6 +62,9 @@ export async function runBuildRefsCli() {
       description: 'Build TypeScript projects',
       flags: {
         boolean: ['clean', 'cache'],
+        default: {
+          cache: true,
+        },
       },
       log: {
         defaultLevel: 'debug',

src/dev/typescript/ref_output_cache/ref_output_cache.ts

@@ -132,7 +132,7 @@ export class RefOutputCache {
       this.log.debug(`[${relative}] clearing outDir and replacing with cache`);
       await del(outDir);
       await unzip(Path.resolve(tmpDir, cacheName), outDir);
-      await Fs.writeFile(Path.resolve(outDir, OUTDIR_MERGE_BASE_FILENAME), archive.sha);
+      await Fs.writeFile(Path.resolve(outDir, OUTDIR_MERGE_BASE_FILENAME), this.mergeBase);
     });
   }
 

x-pack/plugins/case/server/client/cases/mock.ts

@@ -11,6 +11,7 @@ import {
   ConnectorMappingsAttributes,
   CaseUserActionsResponse,
   AssociationType,
+  CommentResponseAlertsType,
 } from '../../../common/api';
 
 import { BasicParams } from './types';
@@ -76,6 +77,20 @@ export const commentAlert: CommentResponse = {
   version: 'WzEsMV0=',
 };
 
+export const commentAlertMultipleIds: CommentResponseAlertsType = {
+  ...commentAlert,
+  id: 'mock-comment-2',
+  alertId: ['alert-id-1', 'alert-id-2'],
+  index: 'alert-index-1',
+  type: CommentType.alert as const,
+};
+
+export const commentGeneratedAlert: CommentResponseAlertsType = {
+  ...commentAlertMultipleIds,
+  id: 'mock-comment-3',
+  type: CommentType.generatedAlert as const,
+};
+
 export const defaultPipes = ['informationCreated'];
 export const basicParams: BasicParams = {
   description: 'a description',

x-pack/plugins/case/server/client/cases/types.ts

@@ -72,7 +72,7 @@ export interface TransformFieldsArgs<P, S> {
 
 export interface ExternalServiceComment {
   comment: string;
-  commentId: string;
+  commentId?: string;
 }
 
 export interface MapIncident {

x-pack/plugins/case/server/client/cases/utils.test.ts

@@ -17,6 +17,8 @@ import {
   basicParams,
   userActions,
   commentAlert,
+  commentAlertMultipleIds,
+  commentGeneratedAlert,
 } from './mock';
 
 import {
@@ -48,7 +50,7 @@ describe('utils', () => {
         {
           actionType: 'overwrite',
           key: 'short_description',
-          pipes: ['informationCreated'],
+          pipes: [],
           value: 'a title',
         },
         {
@@ -71,7 +73,7 @@ describe('utils', () => {
         {
           actionType: 'overwrite',
           key: 'short_description',
-          pipes: ['myTestPipe'],
+          pipes: [],
           value: 'a title',
         },
         {
@@ -98,7 +100,7 @@ describe('utils', () => {
       });
 
       expect(res).toEqual({
-        short_description: 'a title (created at 2020-03-13T08:34:53.450Z by Elastic User)',
+        short_description: 'a title',
         description: 'a description (created at 2020-03-13T08:34:53.450Z by Elastic User)',
       });
     });
@@ -122,13 +124,13 @@ describe('utils', () => {
         },
         fields,
         currentIncident: {
-          short_description: 'first title (created at 2020-03-13T08:34:53.450Z by Elastic User)',
+          short_description: 'first title',
           description: 'first description (created at 2020-03-13T08:34:53.450Z by Elastic User)',
         },
       });
 
       expect(res).toEqual({
-        short_description: 'a title (updated at 2020-03-15T08:34:53.450Z by Another User)',
+        short_description: 'a title',
         description:
           'first description (created at 2020-03-13T08:34:53.450Z by Elastic User) \r\na description (updated at 2020-03-15T08:34:53.450Z by Another User)',
       });
@@ -168,7 +170,7 @@ describe('utils', () => {
       });
 
       expect(res).toEqual({
-        short_description: 'a title (created at 2020-03-13T08:34:53.450Z by elastic)',
+        short_description: 'a title',
         description: 'a description (created at 2020-03-13T08:34:53.450Z by elastic)',
       });
     });
@@ -190,7 +192,7 @@ describe('utils', () => {
       });
 
       expect(res).toEqual({
-        short_description: 'a title (updated at 2020-03-15T08:34:53.450Z by anotherUser)',
+        short_description: 'a title',
         description: 'a description (updated at 2020-03-15T08:34:53.450Z by anotherUser)',
       });
     });
@@ -448,8 +450,7 @@ describe('utils', () => {
           labels: ['defacement'],
           issueType: null,
           parent: null,
-          short_description:
-            'Super Bad Security Issue (created at 2019-11-25T21:54:48.952Z by elastic)',
+          short_description: 'Super Bad Security Issue',
           description:
             'This is a brand new case of a bad meanie defacing data (created at 2019-11-25T21:54:48.952Z by elastic)',
           externalId: null,
@@ -504,15 +505,17 @@ describe('utils', () => {
       expect(res.comments).toEqual([]);
     });
 
-    it('it creates comments of type alert correctly', async () => {
+    it('it adds the total alert comments correctly', async () => {
       const res = await createIncident({
         actionsClient: actionsMock,
         theCase: {
           ...theCase,
           comments: [
             { ...commentObj, id: 'comment-user-1' },
             { ...commentAlert, id: 'comment-alert-1' },
-            { ...commentAlert, id: 'comment-alert-2' },
+            {
+              ...commentAlertMultipleIds,
+            },
           ],
         },
         // Remove second push
@@ -536,14 +539,36 @@ describe('utils', () => {
           commentId: 'comment-user-1',
         },
         {
-          comment:
-            'Alert with ids alert-id-1 added to case (added at 2019-11-25T21:55:00.177Z by elastic)',
-          commentId: 'comment-alert-1',
+          comment: 'Elastic Security Alerts attached to the case: 3',
         },
+      ]);
+    });
+
+    it('it removes alerts correctly', async () => {
+      const res = await createIncident({
+        actionsClient: actionsMock,
+        theCase: {
+          ...theCase,
+          comments: [
+            { ...commentObj, id: 'comment-user-1' },
+            commentAlertMultipleIds,
+            commentGeneratedAlert,
+          ],
+        },
+        userActions,
+        connector,
+        mappings,
+        alerts: [],
+      });
+
+      expect(res.comments).toEqual([
         {
           comment:
-            'Alert with ids alert-id-1 added to case (added at 2019-11-25T21:55:00.177Z by elastic)',
-          commentId: 'comment-alert-2',
+            'Wow, good luck catching that bad meanie! (added at 2019-11-25T21:55:00.177Z by elastic)',
+          commentId: 'comment-user-1',
+        },
+        {
+          comment: 'Elastic Security Alerts attached to the case: 4',
         },
       ]);
     });
@@ -578,8 +603,7 @@ describe('utils', () => {
           description:
             'fun description \r\nThis is a brand new case of a bad meanie defacing data (updated at 2019-11-25T21:54:48.952Z by elastic)',
           externalId: 'external-id',
-          short_description:
-            'Super Bad Security Issue (updated at 2019-11-25T21:54:48.952Z by elastic)',
+          short_description: 'Super Bad Security Issue',
         },
         comments: [],
       });