[Kibana] creating new fields on the fly (Lens, Discover etc..)

[timroes]

In the future we want to rethink index pattern, and one part of index patterns are scripted fields. You can basically specify a Painless script that will be executed instead of a field. So basically that script you specify in index pattern management is inlined into all requests.

I think we want to move that more towards the editor (as nearly everything in index patterns). Also there are a couple of common "scripts" users might want to easily execute without needing to write painless. Let me give you an example: A user might have a field that's looking like 12:18:00 as a string. We don't only want to allow the user to use that field as it is, but also give them the capabilities to e.g. "split that field" by the colon and use then the first one. Splitting fields by special characters is common, so I think we might want to have an option for that in the UI. Behind the scenes that would generate the script for that (return doc['fieldname'].value.split(':')[1] (simplified)) and use that instead of the field (meaning we also must make sure esaggs and esdocs will understand scripts directly instead of fields. We could have common options (like split a field, search and replace a field value, etc.) and have sophisticated safe painless scripts for that as templates that we use behind the scenes, thus the user doesn't need to write a complex painless script (because that version was very simplified, and a sophisticated would have null checks, etc.).

Besides having those scripted templates we also might allow the user to add a custom script directly via the fieldlist (not via the index pattern), since that would allow us to more easily get rid of index patterns later on. We should rather have a way to store those scripts in some kind of library and make them easily reusable later (on any index pattern).

Tags: derived fields / scripted fields / calculated fields

Implementation notes

The first integration will be done using runtime fields saved in the index pattern. Lens provides a way to create new fields and delete/edit existing fields without leaving Lens (using the integration provided here: #88995 ). In a later phase fields saved as part of the Visualization can get added.

[elasticmachine]

Pinging @elastic/kibana-app

[flash1293]

I can see this functionality being super useful for Graph/Discover as well. It probably makes sense to start implementing this as a part of Lens, but we should anticipate making this a generic feature in the future.

[AlonaNadler]

Currently when users ask how to add new fields that didn't exist during ingestions. We usually provide two options:

• General recommendation to add the new field during ingestions
• Scripted fields in Kibana

Two main use case for scripted fields

• Users who want to add a new field after ingestion. Commonly users want the field to become permanent and start being added as part of the ingest
• Users who need a new field for one-time visualization or Discover exploration where it doesn't make sense to add it to the index pattern permanently and for all users of Kibana

Scripted fields allow users to create new fields in the index pattern which do not exists during the time the data is ingested.

Main issues with scripted fields:

• Location: scripted fields exist under management ->index patterns which is not a place regular users are in. It is an area of Kibana which makes it less of a user feature and more an admin feature.
• When scripted field fails it affects all users
• When users load the index pattern it loads the scripted field as well even if the users don’t need it. This behavior makes Kibana feel slower without a real need.
• Painless? - requires the users to know the painless syntax to write a scripted field. Many of the * Kibana users who need this capability don’t know the painless syntax
• The scripted field is always attached to an index pattern. If a user has a need for a scripted field for one time specific visualization. They still need to create it in the index pattern which will be always loaded and available by others for as long as the visualization is needed.
• Performance - although not all users experience scripted field having slower performance. Users sometimes concerned by the performance and prefer not to use scripted fields
• Scripted fields are not available in all Kibana visualizations

How do scripted fields use:

• Normalization - mainly categorization. For example, create a new field review, if the field review_score is >3 review: good , else bad.
• Transformation - “clean” fields to appear more user friendly or just focus on the needed part of the value (unit change, entity extraction, comparison between fields)

Ways in which users would like to use calculated fields/runtime fields:

• Ad-hock per user session - in Discover or when creating visualization ability to create a field on the fly that will be used for specific saved search or visualization. The view in Discover or visualization can be saved to an object that contains the new field or used ad-hoc.
• Enrich in Kibana and move to be part of the ingestions - ability to create a scripted field and at one point move it to be a regular part of the document which will be added during ingestion.
• Proximity to users - whether it is on Discover, Visualize or SIEM, the ability to create the field in place instead of switching context to the management page. Creating a field on the fly in the right context where the users need it. That will also help users find this capability which many or our users are unaware of but really missing it.
• Ability to create a simple field on the fly with easy syntax as users and not admins.

@VijayDoshi

[flash1293]