[APM] Correlations Beta (#86477) #89952

ogupte · 2021-02-02T00:56:47Z

Closes #86477.
Closes #91080.
Closes #90766.
Closes #90622.

x-pack/plugins/apm/public/components/app/Correlations/correlations_table.tsx

elasticmachine · 2021-02-05T23:19:13Z

Pinging @elastic/apm-ui (Team:apm)

x-pack/plugins/apm/public/components/app/Correlations/useFieldNames.ts

sorenlouv · 2021-02-09T20:53:04Z

x-pack/plugins/apm/public/components/app/Correlations/useFieldNames.ts

+  const [defaultFieldNames, setDefaultFieldNames] = useState(
+    getDefaultFieldNames(indexPattern, isRumAgent)
+  );


It feels off that this is stateful. If for perf reasons you can use useMemo.
But first try to strip it down and see if it's needed

It's stateful because useDynamicIndexPatternFetcher is an async fetcher, so it needs to trigger a re-render when the response comes back.

x-pack/plugins/apm/public/components/app/Correlations/custom_fields.tsx

x-pack/plugins/apm/public/components/app/Correlations/useFieldNames.ts

- 'show_correlations_flyout' - 'customize_correlations_fields' - 'select_significant_term'

x-pack/plugins/apm/public/components/shared/search_bar.tsx

sorenlouv · 2021-02-09T23:40:32Z

x-pack/plugins/apm/server/lib/correlations/get_correlations_for_failed_transactions/index.ts

@@ -121,7 +121,7 @@ export async function getErrorRateTimeSeries({
  topSigTerms: TopSigTerm[];
 }) {
  const { start, end, apmEventClient } = setup;
-  const { intervalString } = getBucketSize({ start, end, numBuckets: 30 });
+  const { intervalString } = getBucketSize({ start, end, numBuckets: 15 });


I think this will still return a dynamic number of buckets to aim for a "nice" bucket size.
Some feedback I got was to make the number of buckets fixed.

We'll fix this in a separate issue: #91574

sorenlouv · 2021-02-09T23:47:35Z

...m/server/lib/correlations/get_correlations_for_slow_transactions/get_latency_distribution.ts

+    const buckets = dropRightWhile(
+      distribution.dist_filtered_by_latency.buckets,
+      (bucket, index) => bucket.doc_count === 0 && index > intervalBuckets - 1
+    );


I don't understand why we are doing this. If we want fewer bucket returned shouldn't we specify a larger interval?

Btw. all this should matter less if we use a logarithmic scale. Did we decide to punt on that?

I created a separate issue for the distribution scale: #91574

…tic#91080)

elastic#90622)

ogupte · 2021-02-16T22:11:14Z

jenkins run the e2e

TinaHeiligers · 2021-02-16T22:22:32Z

src/plugins/telemetry/schema/oss_plugins.json

@@ -4384,9 +4384,6 @@
        },
        "apm:enableServiceOverview": {
          "type": "boolean"
-        },
-        "apm:enableCorrelations": {


FYI, You'd need to open a corresponding mapping update request in the infra repo to have this key removed from the telemetry mappings.
If you don't care that it's not there in versions from 7.12, then you won't need to open an issue for that.

We'll probably remove it from the mapping in the infra repo in 7.13. That way we'll be able to tell if users are still using it as a custom UI Setting for version 7.12.

TinaHeiligers

I reviewed both the ui_settings key removal and the usage collection schema update for telemetry.
LGTM

… 15 buckets

ogupte · 2021-02-17T00:10:16Z

jenkins run the e2e

sorenlouv · 2021-02-17T00:21:01Z

x-pack/plugins/apm/server/lib/correlations/get_correlations_for_slow_transactions/index.ts

-                {
-                  range: {
-                    [TRANSACTION_DURATION]: { gte: durationForPercentile },
+              filter: backgroundFilters,


What's the reason for removing the range query for duration? Will removing it not hurt perf since more documents are retrieved?

moving the range query to the function_score provided us with a score that we needed to order foreground results that were most important to us (those with higher durations)

sorenlouv · 2021-02-17T00:21:50Z

x-pack/plugins/apm/server/lib/correlations/get_correlations_for_slow_transactions/index.ts

+                            [TRANSACTION_DURATION]: {
+                              lt: durationForPercentile,
+                            },


Isn't the foreground supposed to be a subset of the background?

... if not you may have to set this

"background_is_superset": false

We found that the existing background filter included too much uninteresting data and it was skewing the results of significant terms agg to be not as helpful see(https://youtu.be/azP15yvbOBA)

sorenlouv · 2021-02-17T00:26:44Z

x-pack/plugins/apm/server/lib/correlations/process_significant_term_aggs.ts

-        bgCount: bucket.bg_count,
-        fgCount: bucket.doc_count,
+        fieldCount: agg.doc_count,
+        valueCount: bucket.doc_count,


Not sure about this change. Can you elaborate a bit?

the previous bgCount wasn't needed since it represents only values for that field/value pair, when we really wanted to express the percentage in terms of the total number of background documents the aggregation applies to. E.g. user_agent.name: HeadlessChrome at the 75th percentile threshold accounts for only 8% of traffic.

previously it showed something like "80% of documents with user_agent.name: HeadlessChrome exist in the 75th percentile threshold", so it's a question of what is easier to understand for the user.

ogupte · 2021-02-17T00:51:55Z

jenkins run the e2e

ogupte · 2021-02-17T01:27:15Z

jenkins run the e2e

ogupte · 2021-02-17T02:07:33Z

@elasticmachine merge upstream

kibanamachine · 2021-02-17T04:20:48Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 50cd63a

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	1731	1733	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	5.2MB	5.2MB	+9.9KB

History

💚 Build #107059 succeeded 7eae6b9
💔 Build #107029 failed cd3407e
💔 Build #107017 failed b5f69ab
💔 Build #107005 failed 0f09c0b
💔 Build #106968 failed cab10a2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [APM] Correlations GA (elastic#86477) * polish and improvements to correlations UI * more improvements and polish * added impact bar * added descriptions * make custom field persistence be unique per service * make custom threshold unique per service in latency correlations * adds telemetry for apm correlations feature. Events: - 'show_correlations_flyout' - 'customize_correlations_fields' - 'select_significant_term' * adds more telemetry for correlations (elastic#90622) * removes the raw score column * replaces experiemental callout with beta badge * replaces threshold number input with percentile option selector * improvements to latency correlations scoring and percentage reporting * removes the 'apm:enableCorrelations' UI setting * - rename useFieldNames.ts -> use_field_names.ts - filter out fields that are not type 'keyword' - feedback improvements * Fixes casing issue for the 'correlations' dir * [APM] Moves correlations button to service details tabslist row (elastic#91080) * [APM] Adds license check for correlations (elastic#90766) * [APM] Adds metrics tracking for correlations views and license prompts (elastic#90622) * Updated the API integration tests to check for new default fields and 15 buckets Co-authored-by: Kibana Machine <[email protected]>

* [APM] Correlations GA (#86477) * polish and improvements to correlations UI * more improvements and polish * added impact bar * added descriptions * make custom field persistence be unique per service * make custom threshold unique per service in latency correlations * adds telemetry for apm correlations feature. Events: - 'show_correlations_flyout' - 'customize_correlations_fields' - 'select_significant_term' * adds more telemetry for correlations (#90622) * removes the raw score column * replaces experiemental callout with beta badge * replaces threshold number input with percentile option selector * improvements to latency correlations scoring and percentage reporting * removes the 'apm:enableCorrelations' UI setting * - rename useFieldNames.ts -> use_field_names.ts - filter out fields that are not type 'keyword' - feedback improvements * Fixes casing issue for the 'correlations' dir * [APM] Moves correlations button to service details tabslist row (#91080) * [APM] Adds license check for correlations (#90766) * [APM] Adds metrics tracking for correlations views and license prompts (#90622) * Updated the API integration tests to check for new default fields and 15 buckets Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

* master: (157 commits) [DOCS] Adds machine learning to the security section of alerting (elastic#91501) [Uptime] Ping list step screenshot caption formatting (elastic#91403) [Vislib] Use timestamp on brush event instead of iso dates (elastic#91483) [Application Usage] Remove deprecated & unused legacy.appChanged API (elastic#91464) Migrate logstash, monitoring, url_drilldowns, xpack_legacy to ts projects (elastic#91194) [APM] Wrap Elasticsearch client errors (elastic#91125) [APM] Fix optimize-tsconfig script (elastic#91487) [Discover][docs] Add searchFieldsFromSource description (elastic#90980) Adds support for 'ip' data type (elastic#85087) [Detection Rules] Add updates from 7.11.2 rules (elastic#91553) [SECURITY SOLUTION] Eql in timeline (elastic#90816) [APM] Correlations Beta (elastic#86477) (elastic#89952) [Security Solutions][Detection Engine] Adds a warning banner when the alerts data has not been migrated yet. (elastic#90258) [Security Solution] [Timeline] Endpoint row renderers (2nd batch) (elastic#91446) skip flaky suite (elastic#91450) skip flaky suite (elastic#91592) [Security Solution][Endpoint][Admin] Endpoint Details UX Enhancements (elastic#90870) [ML] Add better UI support for runtime fields Transforms (elastic#90363) [Security Solution] [Detections] Replace 'partial failure' with 'warning' for rule statuses (elastic#91167) [Security Solution][Detections] Adds Indicator path config for indicator match rules (elastic#91260) ...

ogupte added 3 commits February 5, 2021 16:36

[APM] Correlations GA (elastic#86477)

d321f85

polish and improvements to correlations UI

9110ffa

more improvements and polish

63da7e6

ogupte force-pushed the apm-86477-correlations-ga branch from 82d477e to 63da7e6 Compare February 5, 2021 22:12

ogupte marked this pull request as ready for review February 5, 2021 22:13

ogupte requested a review from a team as a code owner February 5, 2021 22:13

sorenlouv reviewed Feb 5, 2021

View reviewed changes

x-pack/plugins/apm/public/components/app/Correlations/correlations_table.tsx Outdated Show resolved Hide resolved

botelastic bot added the Team:APM All issues that need APM UI Team support label Feb 5, 2021

ogupte added release_note:feature Makes this part of the condensed release notes v7.12.0 labels Feb 6, 2021

ogupte added 4 commits February 9, 2021 11:17

added impact bar

d93f4ca

added descriptions

711bfe6

make custom field persistence be unique per service

38d6a61

make custom threshold unique per service in latency correlations

c535b4a