[Lens] Support the duration formatter #153378

harnish-elastic · 2023-03-21T15:34:25Z

Kibana version: 8.6.0

Elasticsearch version: 8.6.0

Describe the bug:

While migrating visualizations to lens charts, found that lens doesn't have support of convert value to hours/human-readable format which TSVB visualizations have.

In the below screenshot, user can easily see the value into hours.

While migrating TSVB visualizations to Lens, not able to convert into human-readable format. Please refer the below screenshot

elasticmachine · 2023-03-21T20:23:15Z

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

stratoula · 2023-03-22T07:18:55Z

@harnish-elastic can you share with us your configuration in TSVB? Which metric are you using? Do you use a custom formatter?

stratoula · 2023-03-22T07:21:47Z

@drewdaemon why the suffix/decimals settings are not available in the new metric?

harnish-elastic · 2023-03-22T08:30:21Z

@harnish-elastic can you share with us your configuration in TSVB? Which metric are you using? Do you use a custom formatter?

@stratoula Please find the configuration in TSVB. Yes in the TSVB it is using custom formatter which lens doesn't have support. Let me know if you need anything else. Thanks!

Metrics configuration

stratoula · 2023-03-22T08:47:43Z

Got it thanx! I will add this as an enhancement and discuss it with the team

drewdaemon · 2023-03-22T14:49:49Z

why the suffix/decimals settings are not available in the new metric

We allow the standard Intl.NumberFormat API to govern both the suffix and the number of decimal places depending on the number of total digits. I believe the thinking was to make sure

numbers shown in the visualization are a relatively small number of digits (this is a good design principle for single-number visualizations)
internationalization of the units occurs

stratoula · 2023-03-22T14:59:20Z

@drewdaemon I see, but if someone wants to do something like the above then he cant

philippkahr · 2023-03-22T15:01:59Z

@stratoula Please find the configuration in TSVB. Yes in the TSVB it is using custom formatter which lens doesn't have support. Let me know if you need anything else. Thanks!

Why not format the field as a time field using the formatter in the data view and then you have it in Lens, Discover and everywhere always correctly calculated.

stratoula · 2023-03-22T15:03:45Z

~~yes @harnish-elastic you can def do what described above, so no need to support it in Lens. I just wonder if it makes sense to also add this to our new metric.~~
Not possible

philippkahr · 2023-03-22T15:07:51Z

@stratoula I think Kibana should be capable of discovering that this is a long field and then automatically trying to parse if it is a duration and if so, suggest the customer to change this? As a customer knowing to do an "advanced" formatter in a data view isn't that obvious and for all fields that are covered by ECS already we know what data is inside the fields and they should be auto formatted correctly.

stratoula · 2023-03-22T15:24:04Z

Solve it with adding the duration formatter in Lens

harnish-elastic · 2023-03-27T10:08:45Z

@stratoula Please find the configuration in TSVB. Yes in the TSVB it is using custom formatter which lens doesn't have support. Let me know if you need anything else. Thanks!

Why not format the field as a time field using the formatter in the data view and then you have it in Lens, Discover and everywhere always correctly calculated.

Thank you for the solution. We have tried that solution and it works! However, we strongly need this feature into lens chart because we don't want this manual efforts from the user. If the new user comes there is always manual efforts.

lalit-satapathy · 2023-05-18T12:16:00Z

Security team is using formulas to map this currently. Example:

@harnish-elastic., Please confirm if we can do the same short-term?

harnish-elastic · 2023-05-18T13:28:37Z

Security team is using formulas to map this currently. Example: @harnish-elastic., Please confirm if we can do the same short-term?

As per my perspective, if the visualization need only one type of conversion, then we can use formula's. But if we need to convert incrementing values of seconds into min/hours/days/human-readable etc, then we need formatter. For example, as mentioned in the issue, the field is uptime and it's coming in seconds from API and has incremental values. Now if the user's instance uptime value is 8,64,0000 seconds and if we are converting the seconds to hours it will be 2400 hours. While the formatter will give us human-readable values for example 100 days. There could be many scenarios like that. Hence we need duration formatter to convert seconds into human-readable format.

dej611 · 2023-06-13T11:11:45Z

Can we consider this closed by #158468 ?

stratoula · 2023-06-13T11:14:26Z

@dej611 I think the idea here is to add a custom formatter in Lens for the cases that you don't want to use a dataview field formatter as it applied everywhere

dej611 · 2023-06-14T07:22:15Z

With the new custom formatter it is possible to leverage the Numeral.js duration format as well:

Probably it is not as powerful as the moment.js one, but it's good enough probably?

stratoula · 2023-06-14T07:43:10Z

@dej611 we would like something more powerful as TSVB has where you can specify the from to parameters

@MichaelMarcialis

## Summary Fixes #153378 This PR introduces a new Duration formatter, similar to the TSVB one and more complete like the Field Formatter one. Few keys features of the new formatter: * supports from and to unit selection * supports Humanize (approximate) and Humanize (precise) outputs * supports units below milliseconds * supports decimals precision (with the exception of the `humanize approximate` output) * supports suffix * supports the compact version (with the exception of the `humanize approximate` output) Input and Output options are now exported directly from the Field Formatter to have a shared list of options. A first look at the new formatter in action: when enabled it the first time it will lands on `Seconds => Humanize (Approximate)` configuration (this emulates the current Field Formatter behaviour) <img width="1217" alt="Screenshot 2023-07-19 at 15 28 52" src="https://github.com/elastic/kibana/assets/924948/8ddcb945-3cd7-4af1-bdfb-2ae1795ab06c"> In this example the input was in Minutes so it is possible to change it from the default `Seconds` entry: <img width="1223" alt="Screenshot 2023-07-19 at 14 17 11" src="https://github.com/elastic/kibana/assets/924948/b547b3ed-475e-421c-b196-a76b07235bd3"> Switching output type to something else than `Humanize (approximate)` will make available both decimals and compact controls as well. Custom suffix is supported as well. <img width="1220" alt="Screenshot 2023-07-19 at 14 18 34" src="https://github.com/elastic/kibana/assets/924948/f5efe1f7-5f0d-4de1-bb3e-80ff00857bc6"> <img width="1228" alt="Screenshot 2023-07-19 at 14 18 19" src="https://github.com/elastic/kibana/assets/924948/fef7fa2d-ecd3-48ae-a833-16dbeaeaade2"> When Decimals and Compact controls are disabled a helpful tooltip is shown to the user to explain why: <img width="341" alt="Screenshot 2023-07-19 at 14 17 30" src="https://github.com/elastic/kibana/assets/924948/78ca1df5-808d-4e2d-8598-90e3f3ec5a85"> <img width="337" alt="Screenshot 2023-07-19 at 14 17 23" src="https://github.com/elastic/kibana/assets/924948/4ca0028d-3344-464c-8b36-1f74f53bd34e"> I've also prepared an alternate UI version which is more compact, using the `DelimitedControls` UI, but it does not looks that great. Maybe some design iteration could help make it better? cc @MichaelMarcialis <img width="1226" alt="Screenshot 2023-07-19 at 14 20 02" src="https://github.com/elastic/kibana/assets/924948/fb8f69f2-5182-480d-9cb2-7a7149a54320"> ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]>

@MichaelMarcialis

## Summary Fixes elastic#153378 This PR introduces a new Duration formatter, similar to the TSVB one and more complete like the Field Formatter one. Few keys features of the new formatter: * supports from and to unit selection * supports Humanize (approximate) and Humanize (precise) outputs * supports units below milliseconds * supports decimals precision (with the exception of the `humanize approximate` output) * supports suffix * supports the compact version (with the exception of the `humanize approximate` output) Input and Output options are now exported directly from the Field Formatter to have a shared list of options. A first look at the new formatter in action: when enabled it the first time it will lands on `Seconds => Humanize (Approximate)` configuration (this emulates the current Field Formatter behaviour) <img width="1217" alt="Screenshot 2023-07-19 at 15 28 52" src="https://github.com/elastic/kibana/assets/924948/8ddcb945-3cd7-4af1-bdfb-2ae1795ab06c"> In this example the input was in Minutes so it is possible to change it from the default `Seconds` entry: <img width="1223" alt="Screenshot 2023-07-19 at 14 17 11" src="https://github.com/elastic/kibana/assets/924948/b547b3ed-475e-421c-b196-a76b07235bd3"> Switching output type to something else than `Humanize (approximate)` will make available both decimals and compact controls as well. Custom suffix is supported as well. <img width="1220" alt="Screenshot 2023-07-19 at 14 18 34" src="https://github.com/elastic/kibana/assets/924948/f5efe1f7-5f0d-4de1-bb3e-80ff00857bc6"> <img width="1228" alt="Screenshot 2023-07-19 at 14 18 19" src="https://github.com/elastic/kibana/assets/924948/fef7fa2d-ecd3-48ae-a833-16dbeaeaade2"> When Decimals and Compact controls are disabled a helpful tooltip is shown to the user to explain why: <img width="341" alt="Screenshot 2023-07-19 at 14 17 30" src="https://github.com/elastic/kibana/assets/924948/78ca1df5-808d-4e2d-8598-90e3f3ec5a85"> <img width="337" alt="Screenshot 2023-07-19 at 14 17 23" src="https://github.com/elastic/kibana/assets/924948/4ca0028d-3344-464c-8b36-1f74f53bd34e"> I've also prepared an alternate UI version which is more compact, using the `DelimitedControls` UI, but it does not looks that great. Maybe some design iteration could help make it better? cc @MichaelMarcialis <img width="1226" alt="Screenshot 2023-07-19 at 14 20 02" src="https://github.com/elastic/kibana/assets/924948/fb8f69f2-5182-480d-9cb2-7a7149a54320"> ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]>

harnish-elastic added the bug Fixes for quality problems that affect the customer experience label Mar 21, 2023

botelastic bot added the needs-team Issues missing a team label label Mar 21, 2023

harnish-elastic mentioned this issue Mar 21, 2023

[O11y][Apache] Lens Migration elastic/integrations#5617

Closed

4 tasks

jughosta added the Team:Visualizations Visualization editors, elastic-charts and infrastructure label Mar 21, 2023

botelastic bot removed the needs-team Issues missing a team label label Mar 21, 2023

harnish-elastic changed the title ~~[Lens] Not able to convert value into human readable format (Hours)~~ [Lens] Not able to convert value into human readable format ( Seconds / Minutes / Hours ) Mar 22, 2023

harnish-elastic mentioned this issue Mar 22, 2023

[O11y][Hadoop] Migrate application dashboard visualizations to lens in kibana 8.3.0 elastic/integrations#5607

Merged

7 tasks

stratoula added triage_needed and removed bug Fixes for quality problems that affect the customer experience labels Mar 22, 2023

stratoula added enhancement New value added to drive a business result Feature:Lens impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. and removed triage_needed labels Mar 22, 2023

This was referenced Mar 22, 2023

[O11y][Apache] Migrate Access and error logs dashboard visualizations to lens in kibana 8.3.0 elastic/integrations#5635

Merged

[O11y][Apache] Lens Migration elastic/integrations#5608

Closed

kush-elastic mentioned this issue Mar 22, 2023

Lens Migration of OBS integrations elastic/integrations#5221

Closed

36 tasks

harnish-elastic mentioned this issue Mar 22, 2023

[O11y][Apache] Migrate Metrics overview dashboard visualizations to lens in kibana 8.3.0 elastic/integrations#5636

Merged

7 tasks

stratoula changed the title ~~[Lens] Not able to convert value into human readable format ( Seconds / Minutes / Hours )~~ [Lens] Support the duration formatter Mar 22, 2023

timductive added loe:medium Medium Level of Effort impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. and removed impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. labels Mar 22, 2023

This was referenced Apr 4, 2023

[O11y][Hadoop] Remove (s) from Application dashboard visualization elastic/integrations#5731

Merged

[O11y][Redis] Migrate Logs Overview dashboard visualizations to lens in kibana 8.3.0 elastic/integrations#5661

Merged

stratoula mentioned this issue May 29, 2023

[TSVB] Add numeral support for duration fields #14612

Closed

milan-elastic mentioned this issue Jun 6, 2023

[Meta][Lens migration] Phases and Issues tracker elastic/integrations#6489

Closed

5 tasks

dej611 self-assigned this Jul 18, 2023

dej611 mentioned this issue Jul 19, 2023

[Lens] Introduce new duration formatter #162246

Merged

9 tasks

rajvi-patel-22 mentioned this issue Jul 24, 2023

[O11y][Apache] Migrate Uptime metric visualization to lens elastic/integrations#7108

Closed

stratoula closed this as completed in #162246 Jul 24, 2023

This was referenced Jul 24, 2023

[O11y][Hadoop] Update human readable format in Application dashboard elastic/integrations#7112

Closed

[O11y][Redis] Update human readable format in Logs Overview dashboard elastic/integrations#7114

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Support the duration formatter #153378

[Lens] Support the duration formatter #153378

harnish-elastic commented Mar 21, 2023

elasticmachine commented Mar 21, 2023

stratoula commented Mar 22, 2023

stratoula commented Mar 22, 2023

harnish-elastic commented Mar 22, 2023 •

edited

Loading

stratoula commented Mar 22, 2023

drewdaemon commented Mar 22, 2023 •

edited

Loading

stratoula commented Mar 22, 2023

philippkahr commented Mar 22, 2023

stratoula commented Mar 22, 2023 •

edited

Loading

philippkahr commented Mar 22, 2023

stratoula commented Mar 22, 2023

harnish-elastic commented Mar 27, 2023

lalit-satapathy commented May 18, 2023

harnish-elastic commented May 18, 2023

dej611 commented Jun 13, 2023

stratoula commented Jun 13, 2023

dej611 commented Jun 14, 2023

stratoula commented Jun 14, 2023

[Lens] Support the duration formatter #153378

[Lens] Support the duration formatter #153378

Comments

harnish-elastic commented Mar 21, 2023

elasticmachine commented Mar 21, 2023

stratoula commented Mar 22, 2023

stratoula commented Mar 22, 2023

harnish-elastic commented Mar 22, 2023 • edited Loading

stratoula commented Mar 22, 2023

drewdaemon commented Mar 22, 2023 • edited Loading

stratoula commented Mar 22, 2023

philippkahr commented Mar 22, 2023

stratoula commented Mar 22, 2023 • edited Loading

philippkahr commented Mar 22, 2023

stratoula commented Mar 22, 2023

harnish-elastic commented Mar 27, 2023

lalit-satapathy commented May 18, 2023

harnish-elastic commented May 18, 2023

dej611 commented Jun 13, 2023

stratoula commented Jun 13, 2023

dej611 commented Jun 14, 2023

stratoula commented Jun 14, 2023

harnish-elastic commented Mar 22, 2023 •

edited

Loading

drewdaemon commented Mar 22, 2023 •

edited

Loading

stratoula commented Mar 22, 2023 •

edited

Loading