[RAM] [META] Dynamic field selection in Alerts Table

[XavierM]

As user, I should be able to pick any fields from my alert index to show in the alert table as a column like the old alert table.

On the server side:

To be able to do that, we will need to create an internal API to fetch the field from the alert index, by passing a feature id in the request as query params. We will need to make sure that the user has access to this feature so they can access the fields from the index. To achieve that we will do the same thing than here.
Then, you will follow this logic to get the fields through the field capabilities API. We will only do it for o11y since security solution will pass their browser fields as a prop of the alert table. Since we do not want to add description and other attributes to our API, we only to return what indexPatternsFetcherAsInternalUser.getFieldsForWildcard is giving us.

On the front end

We will need to do three things:

1. Add a browserFields prop in our alert table
2. Integrate the browser field component in the bulk action line
3. integrate with our new API to show fields for o11y table like

When this work is done, we will be able to close this issue #133060 and then we will be able to implement a fix to this issue #135101

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[jcger]

Working on the server side part I've seen that the field capabilities api doesn't return the description so I checked how it's done In timelines. Looks like the missing data is added merging the info stored in this 1.49Mb large file. Also, the transformation has this very scary comment. I've copied the file closer to where I need it and implemented a function that only does the merge of the info that I'm missing into the format that the ui component is expecting it to be. Anyways, there are some doubts I was hoping someone could answer:

1. Does the beat_schema/fields file contain the description we'll want to show everywhere the triggers_actions_ui alerts table appears? At least for observability/alerts and security/alerts the descriptions shown are the same but I'm not sure I might be missing somewhere. If so, I'd like to avoid repeating that 1.49Mb file, would there be a fitting common place to move it? It would be used by timelines and rule_registry plugin.
2. The beat_schema/fields is being loaded as part of a search_strategy as you can see here and being registered here. Since I'm not implementing any search strategy, what should be the approach to load this file? For now I just did import it as you can see here but I guess we'd prefer to load it as late as possible.

[kobelb]

@rylnd do you happen to know what https://github.com/elastic/kibana/blob/main/x-pack/plugins/timelines/server/utils/beat_schema/fields.ts is? Is this a list of all ECS fields? The name BeatFields is throwing me off.

[YulNaumenko]

@jcger we are using BeatFields to do the fast enrichment for the description and category for each field in the FieldBrowser component. It's not an issue to move the file to the Kibana package or some other place to share without the circular dependency, but this is a controversial logic, which requires a manual file update and we are willing to change it.

If we take a look closer to the category value - this is the first part of the field name:

Here is the example with the description:

How it is working now:

1. First on the server side we fetch all the fields for one of - dataView or indices list2.
2. Then enrich this fields with category and description from the BeatFields and return this to the client.
3. On the client side we group the fields by category and send is a property to FieldBrowser.

@kobelb ,

Is this a list of all ECS fields?

Not only, ECS - here is basically all the fields from ecsFieldMap and technicalRuleFieldMap

[jcger]

Thank you so much, @YulNaumenko ! The whole controversial part sounds like something that wouldn't be done anytime soon, am I right? If so, I'd like to avoid blocking this task until then. Would it be ok to work with the file as it is, even using it when testing, so we don't forget that it's being used by us and then we'll update once the file is been updated?

One last question, what exactly do you mean with move the file to the Kibana package, I see there is a lot of packages in the packages folder, would it be there? Would it be in the kbn-field-types package?

[YulNaumenko]

@jcger I agree about making the first step by moving the BeatFieds file as it is to the packages.
kbn-field-types doesn't seem right to me, because this if owned by the Application services and contains the different things inside. So, better to create a separate package, which we could remove later by finding some other way for the description/category enrichment.

@spalger maybe you can suggest something about this - is it a good idea or we should find some other way how to share this huge file between security_solution plugin and triggers_actions_ui?

[spalger]

@YulNaumenko Sorry, didn't read everything but my understanding is that there's a massive file that multiple plugins need access to on the front-end. I think it would be cleanest if this file was in a server package and then exposed from route. Assuming the file is in JSON format then fetching the file should be really trivial and not require shared code to make it feasible. If you want to share code for this I'd recommend a second browser-package that exposes a function for fetching the file, parsing it, and giving it the proper types.

The primary benefit of this approach is that we keep the file out of webpack, which makes it easy to bloat the webpack build and slow things down substantially.

[XavierM]

Sorry just came back from vacation, and still catching up. I will like to say that we are not going to use anymore the beat fields, the response ops team does not want to maintain the beat fields file. We decided that we won't show the description of a field for now and we will build the category attribute in our new API. Since we are not going to maintain the description attributes in our alert table, we will let security solution pass their own data to this component so they can maintain the same functionality than before.
We will try in the near future to build an external link to the ECS documentation when we know that this field is part of ECS. We need to figure out how to determine when a field is ECS or not.

[sophiec20]

It is worth clarifying if ECS belongs as an elasticsearch first class citizen - worth having this conversation before we design the implementation?