Merge branch 'master' into ml-remove-angular-filters

docs/discover/kuery.asciidoc

@@ -19,14 +19,13 @@ they appear. This means documents with "quick brown fox" will match, but so will
 to search for a phrase.
 
 The query parser will no longer split on whitespace. Multiple search terms must be separated by explicit
-boolean operators. Note that boolean operators are not case sensitive.
+boolean operators. Lucene will combine search terms with an `or` by default, so `response:200 extension:php` would 
+become `response:200 or extension:php` in KQL. This will match documents where response matches 200, extension matches php, or both. 
+Note that boolean operators are not case sensitive.
 
-`response:200 extension:php` in lucene would become `response:200 and extension:php`.
- This will match documents where response matches 200 and extension matches php.
+We can make terms required by using `and`.
 
-We can make terms optional by using `or`.
-
-`response:200 or extension:php` will match documents where response matches 200, extension matches php, or both.
+`response:200 and extension:php` will match documents where response matches 200 and extension matches php.
 
 By default, `and` has a higher precedence than `or`.
 
@@ -73,7 +72,7 @@ set these terms will be matched against all fields. For example, a query for `re
 in the response field, but a query for just `200` will search for 200 across all fields in your index.
 ============
 
-===== Nested Field Support
+==== Nested Field Support
 
 KQL supports querying on {ref}/nested.html[nested fields] through a special syntax. You can query nested fields in subtly different
 ways, depending on the results you want, so crafting nested queries requires extra thought.
@@ -85,7 +84,8 @@ There are two main approaches to take:
 * *Parts of the query can match different nested documents.* This is how a regular object field works. 
  Although generally less useful, there might be occasions where you want to query a nested field in this way.
 
-Let's take a look at the first approach. In the following document, `items` is a nested field:
+Let's take a look at the first approach. In the following document, `items` is a nested field. Each document in the nested 
+field contains a name, stock, and category.
 
 [source,json]
 ----------------------------------
@@ -116,21 +116,38 @@ Let's take a look at the first approach. In the following document, `items` is a
 }
 ----------------------------------
 
+===== Match a single nested document
+
 To find stores that have more than 10 bananas in stock, you would write a query like this:
 
 `items:{ name:banana and stock > 10 }`
 
-`items` is the "nested path". Everything inside the curly braces (the "nested group") must match a single document. 
-For example, `items:{ name:banana and stock:9 }` does not match because there isn't a single nested document that 
-matches the entire query in the nested group. 
+`items` is the "nested path". Everything inside the curly braces (the "nested group") must match a single nested document. 
+
+The following example returns no matches because no single nested document has bananas with a stock of 9.
+
+`items:{ name:banana and stock:9 }`
+
+==== Match different nested documents
 
-What if you want to find a store with more than 10 bananas that *also* stocks vegetables? This is the second way of querying a nested field, and you can do it like this:
+The subqueries in this example are in separate nested groups and can match different nested documents.
+
+`items:{ name:banana } and items:{ stock:9 }`
+
+`name:banana` matches the first document in the array and `stock:9` matches the third document in the array.
+
+==== Combine approaches
+
+You can combine these two approaches to create complex queries. What if you wanted to find a store with more than 10 
+bananas that *also* stocks vegetables? You could do this:
 
 `items:{ name:banana and stock > 10 } and items:{ category:vegetable }`
 
 The first nested group (`name:banana and stock > 10`) must still match a single document, but the `category:vegetables`
 subquery can match a different nested document because it is in a separate group.
 
+==== Nested fields inside other nested fields
+
 KQL's syntax also supports nested fields inside of other nested fields—you simply have to specify the full path. Suppose you
 have a document where `level1` and `level2` are both nested fields:
 

docs/user/visualize.asciidoc

@@ -24,9 +24,10 @@ To create a visualization:
 . Click on *Visualize* in the side navigation.
 . Click the *Create new visualization* button or the **+** button.
 . Choose the visualization type:
-+
+
 * *Basic charts*
 [horizontal]
+<<lens,Lens>>:: Quickly build several types of basic visualizations by simply dragging and dropping the data fields you want to display.
 <<xy-chart,Line&comma; Area and Bar charts>>:: Compare different series in X/Y charts.
 <<heatmap-chart,Heat maps>>:: Shade cells within a matrix.
 <<pie-chart,Pie chart>>:: Display each source's contribution to a total.
@@ -142,6 +143,8 @@ include::{kib-repo-dir}/visualize/saving.asciidoc[]
 
 include::{kib-repo-dir}/visualize/visualize_rollup_data.asciidoc[]
 
+include::{kib-repo-dir}/visualize/lens.asciidoc[]
+
 include::{kib-repo-dir}/visualize/xychart.asciidoc[]
 
 include::{kib-repo-dir}/visualize/controls.asciidoc[]

docs/visualize/lens.asciidoc

@@ -0,0 +1,186 @@
+[role="xpack"]
+[[lens]]
+== Lens
+
+beta[]
+
+*Lens* provides you with a simple and fast way to create visualizations from your Elasticsearch data. With Lens, you can:
+
+* Quickly build visualizations by dragging and dropping data fields.
+
+* Understand your data with a summary view on each field.
+
+* Easily change the visualization type by selecting the automatically generated visualization suggestions.
+
+* Save your visualization for use in a dashboard.
+
+[float]
+[[drag-drop]]
+=== Drag and drop
+
+The data panel in the left column shows the data fields for the selected time period. When
+you drag a field from the data panel, Lens highlights where you can drop that field. The first time you drag a data field,
+you'll see two places highlighted in green:
+
+* The visualization builder pane
+
+* The *X-axis* or *Y-axis* fields in the right column
+
+You can incorporate many fields into your visualization, and Lens uses heuristics to decide how
+to apply each one to the visualization.
+
+[role="screenshot"]
+image::images/lens_drag_drop.gif[]
+
+TIP: Drag-and-drop capabilities are available only when Lens knows how to use the data. You can still customize
+your visualization if Lens is unable to make a suggestion.
+
+[float]
+[[apply-lens-filters]]
+==== Find the right data
+
+Lens shows you fields based on the <<index-patterns, index patterns>> you have defined in
+{kib}, and the current time range. When you change the index pattern or time filter,
+the list of fields are updated.
+
+To narrow the list of fields you see in the left panel, you can:
+
+* Enter the field name in *Search field names*.
+
+* Click *Filter by type*, then select the filter. You can also select *Only show fields with data*
+to show the full list of fields from the index pattern.
+
+[float]
+[[view-data-summaries]]
+==== Data summaries
+
+To help you decide exactly the data you want to display, get a quick summary of each data field.
+The summary shows the distribution of values in the time range.
+
+To view the data information, navigate to a data field, then click *i*.
+
+[role="screenshot"]
+image::images/lens_data_info.gif[]
+
+[float]
+[[change-the-visualization-type]]
+==== Change the visualization type
+
+With Lens, you are no longer required to build each visualization from scratch. Lens allows
+you to switch between any supported chart type at any time. Lens also provides
+suggestions, which are shortcuts to alternate visualizations based on the data you have.
+
+You can switch between suggestions without losing your previous state:
+
+[role="screenshot"]
+image::images/lens_suggestions.gif[]
+
+If you want to switch to a chart type that is not suggested, click the chart type in the
+top right, then select a chart type. When there is an exclamation point (!)
+next to a chart type, Lens is unable to transfer your current data, but
+still allows you to make the change.
+
+[float]
+[[customize-operation]]
+==== Customize the data for your visualization
+
+Lens allows some customizations of the data for each visualization.
+
+. Change the index pattern.
+
+.. In the left column, click the index pattern name.
+
+.. Select the new index pattern.
++
+If there is a match, Lens displays the new data. All fields that do not match the index pattern are removed.
+
+. Change the data field options, such as the aggregation or label.
+
+.. Click *Drop a field here* or the field name in the right column.
+
+.. Change the options that appear depending on the type of field.
+
+[float]
+[[layers]]
+==== Layers in bar, line, and area charts
+
+The bar, line, and area charts allow you to layer two different series. To add a layer, click *+*.
+
+To remove a layer, click the chart icon next to the index name:
+
+[role="screenshot"]
+image::images/lens_remove_layer.png[]
+
+[float]
+[[lens-tutorial]]
+=== Lens tutorial
+
+Ready to create your own visualization with Lens? Use the following tutorial to create a visualization that
+lets you compare sales over time.
+
+[float]
+[[lens-before-begin]]
+==== Before you begin
+
+To start, you'll need to add the <<add-sample-data, sample ecommerce data>>.
+
+[float]
+==== Build the visualization
+
+Drag and drop your data onto the visualization builder pane.
+
+. Open *Visualize*, then click *Create visualization*.
+
+. On the *New Visualization* window, click *Lens*.
+
+. In the left column, select the *kibana_sample_data_ecommerce* index.
+
+. Click image:images/time-filter-calendar.png[], then click *Last 7 days*. The list of data fields are updated.
+
+. Drag and drop the *taxful_total_price* data field to the visualization builder pane.
++
+[role="screenshot"]
+image::images/lens_tutorial_1.png[Lens tutorial]
+
+Lens has taken your intent to see *taxful_total_price* and added in the *order_date* field to show
+average order prices over time.
+
+To break down your data, drag the *category.keyword* field to the visualization builder pane. Lens
+understands that you want to show the top categories and compare them across the dates,
+and creates a chart that compares the sales for each of the top 3 categories:
+
+[role="screenshot"]
+image::images/lens_tutorial_2.png[Lens tutorial]
+
+[float]
+[[customize-lens-visualization]]
+==== Further customization
+
+Customize your visualization to look exactly how you want.
+
+. In the right column, click *Average of taxful_total_price*.
+
+.. Change the *Label* to `Sales`, or a name that you prefer for the data.
+
+. Click *Top values of category.keyword*.
+
+.. Increase *Number of values* to `10`. The visualization updates in the background to show there are only
+six available categories.
+
+. Look at the suggestions. None of them show an area chart, but for sales data, a stacked area chart
+might make sense. To switch the chart type:
+
+.. Click *Stacked bar chart* in the right column.
+
+.. Click *Stacked area*.
++
+[role="screenshot"]
+image::images/lens_tutorial_3.png[Lens tutorial]
+
+[float]
+[[lens-tutorial-next-steps]]
+==== Next steps
+
+Now that you've created your visualization in Lens, you can add it to a Dashboard.
+
+For more information, see <<dashboard,Dashboard>>.

packages/kbn-es/src/custom_snapshots.js

@@ -24,10 +24,9 @@ function isVersionFlag(a) {
 }
 
 function getCustomSnapshotUrl() {
-  // force use of manually created snapshots until live ones are available
+  // force use of manually created snapshots until ReindexPutMappings fix
   if (!process.env.KBN_ES_SNAPSHOT_URL && !process.argv.some(isVersionFlag)) {
-    // return 'https://storage.googleapis.com/kibana-ci-tmp-artifacts/{name}-{version}-{os}-x86_64.{ext}';
-    return;
+    return 'https://storage.googleapis.com/kibana-ci-tmp-artifacts/{name}-{version}-{os}-x86_64.{ext}';
   }
 
   if (process.env.KBN_ES_SNAPSHOT_URL && process.env.KBN_ES_SNAPSHOT_URL !== 'false') {

src/core/MIGRATION.md

@@ -1139,7 +1139,7 @@ import { setup, start } from '../core_plugins/visualizations/public/legacy';
 | `ui/embeddable`                                   | `embeddables`                                                | still in progress                                                                                                                                            |
 | `ui/filter_manager`                               | `data.filter`                                                | --                                                                                                                                                           |
 | `ui/index_patterns`                               | `data.indexPatterns`                                         | still in progress                                                                                                                                            |
-| `ui/registry/feature_catalogue                    | `feature_catalogue.register`                                 | Must add `feature_catalogue` as a dependency in your kibana.json.                                                                                            |
+| `ui/registry/feature_catalogue`                   | `home.featureCatalogue.register`                             | Must add `home` as a dependency in your kibana.json.                                                                                            |
 | `ui/registry/vis_types`                           | `visualizations.types`                                       | --                                                                                                                                                           |
 | `ui/vis`                                          | `visualizations.types`                                       | --                                                                                                                                                           |
 | `ui/share`                                        | `share`                                                      | `showShareContextMenu` is now called `toggleShareContextMenu`, `ShareContextMenuExtensionsRegistryProvider` is now called `register`                         |
@@ -1182,7 +1182,7 @@ This table shows where these uiExports have moved to in the New Platform. In mos
 | `fieldFormatEditors`         |                                                                                                                           |                                                                                                                                       |
 | `fieldFormats`               |                                                                                                                           |                                                                                                                                       |
 | `hacks`                      | n/a                                                                                                                       | Just run the code in your plugin's `start` method.                                                                                    |
-| `home`                       | [`plugins.feature_catalogue.register`](./src/plugins/feature_catalogue)                                                   | Must add `feature_catalogue` as a dependency in your kibana.json.                                                                     |
+| `home`                       | [`plugins.home.featureCatalogue.register`](./src/plugins/home/public/feature_catalogue)                                                    | Must add `home` as a dependency in your kibana.json.                                                                     |
 | `indexManagement`            |                                                                                                                           | Should be an API on the indexManagement plugin.                                                                                       |
 | `injectDefaultVars`          | n/a                                                                                                                       | Plugins will only be able to "whitelist" config values for the frontend. See [#41990](https://github.com/elastic/kibana/issues/41990) |
 | `inspectorViews`             |                                                                                                                           | Should be an API on the data (?) plugin.                                                                                              |

src/core/public/mocks.ts

@@ -117,9 +117,22 @@ function createCoreContext(): CoreContext {
   };
 }
 
+function createStorageMock() {
+  const storageMock: jest.Mocked<Storage> = {
+    getItem: jest.fn(),
+    setItem: jest.fn(),
+    removeItem: jest.fn(),
+    clear: jest.fn(),
+    key: jest.fn(),
+    length: 10,
+  };
+  return storageMock;
+}
+
 export const coreMock = {
   createCoreContext,
   createSetup: createCoreSetupMock,
   createStart: createCoreStartMock,
   createPluginInitializerContext: pluginInitializerContextMock,
+  createStorage: createStorageMock,
 };

src/core/server/saved_objects/service/lib/filter_utils.test.ts

@@ -405,5 +405,29 @@ describe('Filter Utils', () => {
         },
       ]);
     });
+
+    test('Return Error if filter is using an non-existing key null key', () => {
+      const validationObject = validateFilterKueryNode(
+        fromKueryExpression('foo.attributes.description: hello AND bye'),
+        ['foo'],
+        mockMappings
+      );
+      expect(validationObject).toEqual([
+        {
+          astPath: 'arguments.0',
+          error: null,
+          isSavedObjectAttr: false,
+          key: 'foo.attributes.description',
+          type: 'foo',
+        },
+        {
+          astPath: 'arguments.1',
+          error: 'The key is empty and needs to be wrapped by a saved object type like foo',
+          isSavedObjectAttr: false,
+          key: null,
+          type: null,
+        },
+      ]);
+    });
   });
 });