[lens] Show field details on click (elastic#44697)

* [lens] Initial Commit (elastic#35627)

* [visualization editor] Initial Commit

* [lens] Add more complete initial state

* [lens] Fix type issues

* [lens] Remove feature control

* [lens] Bring back feature control and add tests

* [lens] Update plugin structure and naming per comments

* replace any usage by safe casting

* [lens] Respond to review comments

* [lens] Remove unused EditorFrameState type

* [lens] Initial state for IndexPatternDatasource (elastic#36052)

* [lens] Add first tests to indexpattern data source

* Respond to review comments

* Fix type definitions

* [lens] Editor frame initializes datasources and visualizations (elastic#36060)

* [lens] Editor frame initializes datasources and visualizations

* Respond to review comments

* Fix build issues

* Fix state management issue

* [lens][draft] Lens/drag drop (elastic#36268)

Add basic drag / drop component to Lens

* remove local package (elastic#36456)

* [lens] Native renderer (elastic#36165)

* Add nativerenderer component

* Use native renderer in app and editor frame

* [Lens] No explicit any (elastic#36515)

* [Lens] Implement basic editor frame state handling (elastic#36443)

* [lens] Load index patterns and render in data panel (elastic#36463)

* [lens] Editor frame initializes datasources and visualizations

* Respond to review comments

* Fix build issues

* remove local package

* [lens] Load index patterns into data source

* Redo types for Index Pattern Datasource

* Fix one more type

* Respond to review comments

* [draft] Lens/line chart renderer (elastic#36827)

Expression logic for the Lens xy chart.

* [lens] Index pattern data panel (initial) (elastic#37015)

* [lens] Index pattern switcher

* Respond to review comments

* [Lens] Editor state 2 (elastic#36513)

* [lens] Dimension panel that generates columns (elastic#37117)

* [lens] Dimension panel that generates columns

* Update from review comments

* [lens] Generate esdocs queries from index pattern (elastic#37361)

* [lens] Generate esdocs queries from index pattern

* Remove unused code

* Update yarn.lock from yarn kbn bootstrap

* [Lens] Add basic Lens xy chart suggestions (elastic#37030)

Basic xy chart suggestions

* [Lens] Expression rendering (elastic#37648)

* [Lens] Expression handling (elastic#37876)

* [Lens] Lens/xy config panel (elastic#37391)

Basic xy chart configuration panel

* [Lens] Xy expression building (elastic#37967)

* [Lens] Initialize visualization with datasource api (elastic#38142)

* [lens] Dimension panel lets users select operations and fields individually (elastic#37573)

* [lens] Dimension panel lets users select operations and fields individually

* Split files and add tests

* Fix dimension labeling and add clear button

* Support more aggregations, aggregation nesting, rollups, and clearing

* Fix esaggs expression

* Increase top-level test coverage of dimension panel

* Update from review comments

* [Lens] Rename columns (elastic#38278)

* [Lens] Lens/index pattern drag drop (elastic#37711)

* Basic xy chart suggestions

* Re-apply XY config panel after force merge

* Initial integration of lens drag and drop

* Tweak naming, remove irellevant comment

* Tweaks per Wylie's feedback

* Add xy chart internationalization
Tweak types per Joe's feedback

* Update xy chart i18n implementation

* Fix i18n id

* Add drop tests to the lens index pattern

* improve tests

* [lens] Only allow aggregated dimensions (elastic#38820)

* [lens] Only allow aggregated dimensions

* [lens] Index pattern suggest on drop

* Fully remove value

* Revert "[lens] Index pattern suggest on drop"

This reverts commit 604c6ed.

* Fix type errors

* [lens] Suggest on drop (elastic#38848)

* [lens] Index pattern suggest on drop

* Add test for suggestion without date field

* fix merge

* [Lens] Parameter configurations and new dimension config flow (elastic#38863)

* fix eslint failure

* [lens] Fix build by updating saved objects and i18n (elastic#39391)

* [lens] Update location of saved objects code

* Update internatationalization

* Remove added file

* [lens] Fix arguments to esaggs using booleans (elastic#39462)

* [lens] Datatable visualization plugin (elastic#39390)

* [lens] Datatable visualization plugin

* Fix merge issues and add tests

* Update from review

* Fix file locations

* [lens] Use first suggestion when switching visualizations (elastic#39377)

* [lens] Label each Y axis with its operation label (elastic#39461)

* [lens] Label each Y axis with its operation label

* Remove comment

* Add link to chart issue

* [Lens] Suggestion preview rendering (elastic#39576)

* [Lens] Popover configs (elastic#39565)

* [Lens] Basic layouting (elastic#39587)

* remove datasource public API in suggestions (elastic#39772)

* [Lens] Basic save / load (elastic#39257)

Add basic routing, save, and load to Lens

* [lens] Fix lint error

* [lens] Use node scripts/eslint.js --fix to fix errors

* [lens] Include link to lens from Visualize (elastic#40542)

* [lens] Support stacking in xy visualization (elastic#40546)

* [lens] Support stacking in xy visualization

* Use chart type switcher for stacked and horizontal xy charts

* Clean up remaining isStacked code

* Fix type error

* [Lens] Add xy split series support (elastic#39726)

* Add split series to lens xy chart

* [lens] Lens Filter Ratio (elastic#40196)

* WIP filter ratio

* Fix test issues

* Pass dependencies through plugin like new platform

* Pass props into filter ratio popover editor

* Provide mocks to filter_ratio popover test

* Add another test

* Clean up to prepare for review

* Clean up unnecessary changes

* Respond to review comments

* Fix tests

* [Lens] Terms order direction (elastic#39884)

* fix types

* [Lens] Data panel styling and optimizations (elastic#40787)

Style the data panel (mostly Joe Reuter's doing). Optimize a bunch of the Lens stack.

* [Lens] Optimize dimension panel flow (elastic#41114)

* [Lens] re-introduce no-explicit-any (elastic#41454)

* [Lens] No results marker (elastic#41450)

* [lens] Support layers for visualizing results of multiple queries (elastic#41290)

* [lens] WIP add support for layers

* [lens] WIP switch to nested tables

* Get basic layering to work

* Load multiple tables and render in one chart

* Fix priority ordering

* Reduce quantity of linting errors

* Ensure that new xy layer state has a split column

* Make the "add" y / split the trailing accessor

* Various fixes for datasource public API and implementation

* Unify datasource deletion and accessor removal

* Fix broken scss

* Fix xy visualization TypeScript errors?

* Build basic suggestions

* Restore save/load and fix typescript bugs

* simplify init routine

* fix save tests

* fix persistence tests

* fix state management tests

* Ensure the data table is aligned to the top

* Add layer support to Lens datatable

* Give xy chart a default layer initially

* Allow deletion of layers in xy charts

* xy: Make split accessor singular
Remove commented code blocks

* Change expression type for lens_merge_tables

* Fix XY chart rendering expression

* Fix type errors relating to `layerId` in table suggestions

* Pass around tables for suggestions with associated layerIds

* fix tests in workspace panel

* fix editor_frame tests

* Fix xy tests, skip inapplicable tests
that will be implemented in a separate PR

* add some tests for multiple datasources and layers

* Suggest that split series comes before X axis in XY chart

* Get datatable suggestion working

* Adjust how xy axis labels are computed

* Datasource suggestions should handle layers and have tests

* Fix linting in XY chart and remove commented code

* Update snapshots from earlier change

* Fix linting errors

* More cleanup

* Remove commented code

* Test the multi-column editor

* XY Visualization does not need to track datasourceId

* Fix various comments

* Remove unused xy prop
Add datasource header to datatable config

* Use operation labels for XY chart

* Adding and removing layers is reflected in the datasource

* rewrote datasource state init

* clean up editor_frame frame api implementation

* clean up editor frame

* [Lens] Embeddable (elastic#41361)

* [lens] Move XY chart config into popover and fix layering (elastic#41927)

* [lens] Move XY chart config into popover and fix layering

* Fix tests

* Update style

* Change wrapping of layer settings popover

* [Lens] Fix bugs in date_histogram and filter ratio (elastic#42046)

* [Lens] Performance improvements (elastic#41784)

* fix type error

* switch default size of terms operation to 3 (elastic#42334)

* [lens] Improve suggestions for split series (elastic#42052)

* [lens] Add chart switcher (elastic#42093)

* solve merge conflicts

* fix test case

* [Lens] Allow only current visualization on field drop in workspace (elastic#42344)

* [Lens] Remove indexpattern id on column (elastic#42429)

* [lens] Implement app-level filtering and time picker (elastic#42031)

* [lens] Implement app-level filtering and time picker

* More integration with filter bar

* Clean up test code and type errors

* Add frame level tests for syncing with app

* Add test coverage for app logic

* Simplify state management from app down

* Fix import errors

* Clarify whether properties are ids or titles for index pattern

* pass new saved object by ref

* add dirty state checking

* Fix tests

* [Lens] Add some tests around document handling in dimension panel (elastic#42670)

* [Lens] Terms operation boolean support (elastic#42817)

* [lens] Minor UX/UI improvements in Lens (elastic#42852)

* Make dimension popover toggle when clicking button

* Without suggestions hide suggestion panel

* Add missing translations (elastic#42921)

* [Lens] Config panel design (elastic#42980)

* Fix up design of config panel

Does not include config popover

* Remove a couple of non-null assertions (elastic#43013)

* Remove a couple of non-null assertions

* Remove orphaned import

* [Lens] Switch indexpattern manually (elastic#42599)

* [Lens] Update frame to put suggestions at the bottom (elastic#42997)

* fix type errors

* switch indexpattern on layer if there is only a single empty one (elastic#43079)

* [Lens] Suggest reduced versions of current data table (elastic#42537)

* [Lens] Field formatter support (elastic#38874)

* Fix bugs

* [Lens] Add bucket nesting editor to indexpattern (elastic#42869)

* [Lens] Remove unnecessary fields and indexing from mappings (elastic#43285)

* [Lens] Xy scale type (elastic#42142)

* [lens] Allow updater function to be used for updating state (elastic#43373)

* [Lens] Lens metric visualization (elastic#39364)

* Fix axis rotation (elastic#43792)

* [Lens] Auto date histogram (elastic#43775)

* Add auto date histogram

* Improve documentation and cleanup

* Add tests

* Change test name

* [Lens] Fix query bar integration (elastic#43865)

* [Lens] Clean up operations code (elastic#43784)

* [Lens] Functional tests (elastic#44279)

Foundational layer for lens functional tests. Lens-specific page objects are not in this PR.

* [Lens] Add Lens visualizations to Visualize list (elastic#43398)

* [Lens] Suggestion improvements (elastic#43688)

* [lens] Calculate existence of fields in datasource

* [lens] Show field details on click

* [lens] Calculate existence of fields in datasource (elastic#44422)

* [lens] Calculate existence of fields in datasource

* Fix route registration

* Add page object and use existence in functional test

* Simplify layout of filters for index pattern

* Respond to review feedback

* Update class names

* Use new URL constant

* Fix usage of base path

* Fix lint errors

* Add basic API test and make trigger into draggable

* [Lens ] Preview metric (elastic#43755)

* format filter ratio as percentage (elastic#44625)

* [Lens] Remove datasource suggestion id (elastic#44495)

* [Lens] Make breadcrumbs look and feel like Visualize (elastic#44258)

* Improve click target for popover

* [lens] Fix breakage from app-arch movements (elastic#44720)

* [lens] Fix type error in test from merge

* Merge branch origin/feature/lens

* [lens] Fix registration of embeddable (elastic#45171)

* Upgrade EUI and fix typescript breakages

* Undo EUI changes

* [Lens] Functional tests (elastic#44814)

Basic functional tests for Lens, by no means comprehensive. This is more of a smokescreen test of some normal use cases.

* Fix snapshots

* Fix issues with types and tests

* Add back accidentally committed code

* [lens] Add Lens to CODEOWNERS (elastic#45296)

* Change popover styles and simplify types on server

* [lens] Fix visualization alias registration

* [lens] Fix usage of EUI after typescript upgrade (elastic#45404)

* [lens] Fix usage of EUI after typescript upgrade

* Use local fix instead of workaround

* Process results more

* Fix API errors

* Fix API tests and use sampleCount

* Simplify document count calculation

* [lens] Fix usage of expressions plugin (elastic#45544)

* [lens] Fix usage of expressions plugin

* Use updated exports from elastic#45538

* Fix imports and mocha tests

* Use relative instead of absolute path to fix tests

* Fix API test

* Fix type errors and pass through new platform from top

* [lens] More cleanup from QueryBar changes in master (elastic#45687)

* Fix tests

* Fix i18n issue

* [lens] Fix build and use new platform from entry points (elastic#45834)

* [lens] Fix build and use new platform from entry points

* Fix params for existence route

* Fix tests and add boolean/empty field support

* Updated top values

* Fixed up overall layout including adding a popover header

* Fix up time display

* Improve popover edge cases and respond to feedback

* Fix field formatters for histogram

* Use custom style for empty top values

x-pack/legacy/plugins/apm/typings/elasticsearch.ts

@@ -28,7 +28,9 @@ declare module 'elasticsearch' {
     | 'extended_stats'
     | 'filter'
     | 'filters'
-    | 'cardinality';
+    | 'cardinality'
+    | 'sampler'
+    | 'value_count';
 
   type AggOptions = AggregationOptionMap & {
     [key: string]: any;
@@ -71,6 +73,12 @@ declare module 'elasticsearch' {
     >;
   };
 
+  type SamplerAggregation<SubAggregationMap> = SubAggregation<
+    SubAggregationMap
+  > & {
+    doc_count: number;
+  };
+
   interface AggregatedValue {
     value: number | null;
   }
@@ -82,7 +90,9 @@ declare module 'elasticsearch' {
         max: AggregatedValue;
         min: AggregatedValue;
         sum: AggregatedValue;
-        terms: BucketAggregation<AggregationOption[AggregationName]>;
+        value_count: AggregatedValue;
+        // Elasticsearch might return terms with numbers, but this is a more limited type
+        terms: BucketAggregation<AggregationOption[AggregationName], string>;
         date_histogram: BucketAggregation<
           AggregationOption[AggregationName],
           number
@@ -128,6 +138,7 @@ declare module 'elasticsearch' {
         cardinality: {
           value: number;
         };
+        sampler: SamplerAggregation<AggregationOption[AggregationName]>;
       }[AggregationType & keyof AggregationOption[AggregationName]];
     }
   >;

x-pack/legacy/plugins/lens/common/api.ts

@@ -0,0 +1,36 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export interface BucketedAggregation<KeyType = string> {
+  buckets: Array<{
+    key: KeyType;
+    count: number;
+  }>;
+}
+
+export interface NumberStatsResult {
+  count: number;
+  histogram: BucketedAggregation<number>;
+  topValues: BucketedAggregation<number>;
+}
+
+export interface TopValuesResult {
+  count: number;
+  topValues: BucketedAggregation<string>;
+}
+
+export interface FieldStatsResponse<KeyType = unknown> {
+  // Total count of documents
+  totalDocuments?: number;
+  // If sampled, the exact number of matching documents
+  sampledDocuments?: number;
+  // If sampled, the exact number of values sampled. Can be higher than documents
+  // because Elasticsearch supports arrays for all fields
+  sampledValues?: number;
+  // Histogram and values are based on distinct values, not based on documents
+  histogram?: BucketedAggregation<KeyType>;
+  topValues?: BucketedAggregation<KeyType>;
+}

x-pack/legacy/plugins/lens/common/index.ts

@@ -4,4 +4,5 @@
  * you may not use this file except in compliance with the Elastic License.
  */
 
+export * from './api';
 export * from './constants';

x-pack/legacy/plugins/lens/public/indexpattern_plugin/_field_itm.scss

@@ -0,0 +1,21 @@
+.lnsFieldItem__topValue {
+  margin-bottom: $euiSizeS;
+
+  &:last-of-type {
+    margin-bottom: 0;
+  }
+}
+
+.lnsFieldItem__topValueProgress {
+  background-color: $euiColorLightestShade;
+
+  &::-webkit-progress-bar {
+    background-color: $euiColorLightestShade;
+  }
+}
+
+.lnsFieldItem__fieldPopoverPanel {
+  min-width: 260px;
+  max-width: 300px;
+}
+

x-pack/legacy/plugins/lens/public/indexpattern_plugin/datapanel.tsx

@@ -532,11 +532,14 @@ export const InnerIndexPatternDataPanel = function InnerIndexPatternDataPanel({
                 const overallField = fieldByName[field.name];
                 return (
                   <FieldItem
+                    core={core}
                     indexPattern={currentIndexPattern}
                     key={field.name}
                     field={field}
                     highlight={localState.nameFilter.toLowerCase()}
                     exists={overallField ? !!overallField.exists : false}
+                    dateRange={dateRange}
+                    query={query}
                   />
                 );
               })}