Expanding Security Integration Categories

[jamiehynds]

Currently we have a very broad 'security' tag for integrations in both Fleet and our public facing integrations page. As we continue to expand the security data sources we support, this approach doesn't scale and will lead to discoverability issues when users are trying to find relevant integrations.

I'm proposing that we create new categories that are narrow in scope and will have several integrations associated with each tag, as we build new integrations.

Proposed categories - open to all feedback/suggestions:

• Endpoint Security
• Email
• Firewalls
• Identity and Access Management
• Threat Intelligence
• Web Application Firewall (WAF)
• Vulnerability

[kaiyan-sheng]

I created an issue about AWS package categories #1342 which is related to this main topic.

[shimonmodi]

Thanks for putting the list together @jamiehynds - this makes sense. As the PM for threat intelligence I would absolutely advocate for it to be separate category. It serves a specific purpose, the data is structured to reflect specific use cases, end users recognize it as a standalone capability in SecOps and vendors categorize themselves as threat intel vendors. A more fine grained categorization will also improve the end user's experience.

[jamiehynds]

@mostlyjason while more discussion is needed to finalize these new security related categories, do you envisage any issues on the Fleet side with adding new categories?

@akshay-saraswat does the o11y team have a need for new integration categories? if so, maybe we could align on timing and update the categories in one go.

[mostlyjason]

I worry that we'll have too many categories. According to fullstory, the median page fold is around 11 categories so most browsers cannot see the whole list above the fold. Also, we are planning to add a bunch of new non-agent integrations which may expand the categories elastic/kibana#93084. The more categories, the longer it will take users to read/understand them all. That reduces the value of the categories as a quick way for users to find what they are looking for.

As an alternative, we are planning to add more keyword support so someone searching on "firewall" could see sophos, cisco, etc. You could also add keywords like "threat intelligence" elastic/package-spec#209

Should we look at the list holistically and try to remove some less valuable ones to make space? For example, Monitoring only has 3 integrations despite being a broad term. Config mgmt only has 2 integrations. How many would be in the new set of categories proposed here?

[jamiehynds]

Thanks for the additional context Jason - I hadn't considered the impact of the fold/scroll if we expand the categories. Although, we definitely have a need for additional categories on the security side, to avoid the situation whereby 50+ integrations exist under security today.

The keyword support would be a great addition- how would we, on the integrations side, specify the keywords associated with an integration?

A review of the existing categories and associated integrations is a good start too. As an example, I originally proposed 'Collaboration' as a new category, but 'Productivity' exists, so will go with that. To give you an idea of the future integrations per category, here's our 'new integrations' board: https://github.com/orgs/elastic/projects/532

[mostlyjason]

@jamiehynds are you still the security representative for ECS? We previously talked about making category changes through an ECS-like process for governance. I imagine you're more familiar with that process than I am. Do you think its a good fit to give us the more holistic perspective and provide a way to align multiple teams on a shared set of categories? If so, would you want to take the lead on defining that process?

It'd be good to align on process/ownership with @akshay-saraswat as well since he is the PM for the ecosystem team. I'm focused more on the Integrations UI so my concern is mainly from the design perspective and to make sure we are all communicating/coordinating as a company.

[jamiehynds]

@mostlyjason we looked at categorising data sources in ECS awhile back, but didn't get too far. Can certainly revisit it though. Given that most of the ECS categorisation fields are based on 'events', I'm thinking an integration.* field set may be valuable, especially as we embrace vendor developed integrations. We could have integration.category, integration.version, integration.built_by, integration.support_by, etc. Will create an RFC and discuss from there.

Totally understand the concern on the UI side. Will work through ECS and agreed upon categories, then revisit the UI aspect once we have a defined set.

[dhru42]

@jamiehynds - is the goal to make a parent Security tag under which there are categories (i.e. endpoint security, threat intelligence, etc.)?

[gavinwye]

Some thoughts on this from a design perspective.

Page fold and scrolling
I'm not really concerned about that. The low scroll depth could be explained by the search bar at the top of the page. Adding more categories/links puts the content on the page that way users have a chance to find it. It's a lot easier to scroll than it is to click a link. When searching you have to know what you're looking for.

Reorganising the navigation
Thinking about adding links/categories it would be best to look at this holistically. Some questions:

• What are the most frequently used integrations people use? We should look to make these the most findable.
• Are there integrations that get added as a group? Can we group these together, possibly even add them together?

If I was going to tackle this I'd set up a remote card sort using something like Optimal sort

I'd also want to look at the design of the page as a whole I think there are things that could be done to make it more useful to users. Is anyone working on this from a design perspective?

[akshay-saraswat]

Please review this proposal doc and follow a process to make sure that these categories are actually required and not going to hurt our UX in the long run.

[maxcold]

For the context: Threat Intelligence category has been discussed and added in the context of this issue elastic/package-spec#222

[jamiehynds]

Closing via #5123