Track Newly Released Integrations per Elastic Release in the Docs

[BenB196]

Enhancement

Currently Elastic releases new integrations on just about every non-patch release. However, an issue here is that there isn't a good way for a user to find out which integrations are added in which version. So, if they're trying to find out what new things were added in a release, or if they are trying to determine which release, they need to upgrade to, to get an integration, it requires a lot of digging within GitHub.

It'd be nice if this could be tracked somewhere in the docs, so users don't need to dig around for this information.

Related discussion thread: https://discuss.elastic.co/t/where-to-find-a-list-of-newly-added-integrations-for-per-release/291363

[mtojek]

Hey @BenB196,

it isn't the final solution, but a workaround that can help you list integrations in rather human-readable way:

curl "https://epr.elastic.co/search?packageName=aws&kibana.version=7.16.0" -s | jq '.[] | (.name + "-" + .version)'

This command will list packages compatible with Kibana v7.16.0. If you want to check compatibility against a different stack version, just replace 7.16.0 with something else, for example8.0.0.

[jamiehynds]

@bmorelli25 is this something we could include as part of our integration docs? A section such as 'Release History' would suffice, showing the last 3(?) stack releases and the integrations that were shipped as part of that release?

[bmorelli25]

A section such as 'Release History' would suffice, showing the last 3(?) stack releases and the integrations that were shipped as part of that release?

That's a good idea @jamiehynds. This is essentially prettifying the output of Marcin's command above and adding it to the docs? I don't see why not.

Note that I'm also working on getting complete release notes added to individual integration doc pages right now.

[bmorelli25]

Does it make sense to also include release information? Something like this (with a bit more cleanup still needed)

curl "https://epr.elastic.co/search?packageName=aws&kibana.version=7.16.0" -s | jq '.[] | ("| " + .title + " | " + .version + " | " + .release + " | ")' -r

Output...

Package	Version	Release status
1Password Events Reporting	0.2.0	beta
AWS	1.6.1	ga
ActiveMQ	0.3.0	beta
Apache HTTP Server	1.3.2	ga
Apache Tomcat	1.2.0	ga
Auditd	1.3.0	ga
Azure Application Insights Metrics Overview	0.3.0	beta
Azure Billing Metrics	0.2.0	beta
Azure Logs	0.12.0	beta
Azure resource metrics	0.3.2	beta
CEF Logs	1.3.0	ga
Cassandra	1.1.0	ga
Check Point	1.2.0	ga
Cisco ASA	1.3.0	ga
Cisco Duo	0.2.0	beta
Cisco FTD	1.2.0	ga
Cisco IOS	1.2.0	ga
Cisco Secure Endpoint (AMP)	0.2.0	beta
Cloudflare	1.1.0	ga
CockroachDB Metrics	0.2.0	beta
CrowdStrike Logs	1.1.0	ga
Custom TCP Logs	1.0.0	ga
Custom UDP Logs	1.0.0	ga
Custom Windows Event Logs	1.2.0	ga
CyberArk Privileged Access Security Logs	2.2.0	ga
Docker	1.0.0	ga
Elastic APM	7.16.0	ga
Elastic Agent	1.3.0	ga
Elastic Synthetics	0.5.0	beta
Endpoint Security	1.2.2	ga
Fireeye	1.1.0	ga
Fleet Server	1.1.0	ga
Fortinet Logs	1.3.0	ga
Google Cloud Platform	1.2.0	ga
Google Santa Logs	1.1.0	ga
Google Workspace Audit Reports	1.2.0	ga
Hashicorp Vault	1.2.0	ga
IIS	0.8.0	beta
Juniper Logs	1.0.4	ga
Kafka	1.1.0	ga
Keycloak	1.1.0	ga
Kibana	1.0.0	ga
Kubernetes	1.5.0	ga
Linux	0.4.1	beta
Logstash	1.0.0	ga
Microsoft	1.0.0	ga
Microsoft DHCP	0.2.0	beta
Microsoft Defender for Endpoint	1.1.0	ga
MongoDB	1.2.0	ga
MySQL	1.1.0	ga
NATS	1.2.0	ga
NetFlow Records	1.3.0	ga
Nginx	1.2.1	ga
Nginx Ingress Controller Logs	1.2.0	ga
Office 365 Logs	1.3.0	ga
Okta Logs	1.3.0	ga
Osquery Logs	1.1.0	ga
Osquery Manager	0.8.0	ga
Palo Alto Cortex XDR Logs	0.2.4	beta
Palo Alto Networks Logs	1.3.0	ga
PostgreSQL	1.2.0	ga
Prebuilt Security Detection Rules	0.14.3	ga
RabbitMQ Logs	1.2.0	ga
Redis	1.1.0	ga
STAN	1.2.0	ga
Sophos Logs	1.1.0	ga
Suricata Events	1.3.0	ga
System	1.6.4	ga
Tenable.sc	0.1.0	beta
Traefik	1.2.0	ga
VMware vSphere	0.1.0	beta
Windows	1.5.0	ga
Zeek Logs	1.5.0	ga
ZeroFox	1.1.0	ga
ZooKeeper Metrics	1.2.0	ga
Zoom	1.1.0	ga

[BenB196]

From my perspective a prettier/cleaner version of:

diff -u <(curl "https://epr.elastic.co/search?packageName=aws&kibana.version=7.15.1" -s | jq '.[] | (.name + "-" + .version)') <(curl "https://epr.elastic.co/search?packageName=aws&kibana.version=7.16.1" -s | jq '.[] | (.name + "-" + .version)') | sort

Would be useful from a change perspective to see what actually was added/removed/updated. Since the new unified integration tab in Kibana in 7.16, I found it hard to discover what actually happened with integrations since 7.15.

A bit of a complex solution. would be to have a table in the docs, which would allow you to select two version of the stack, then show the diff in integrations between the 2 versions.

[jamiehynds]

sorry for the delay @bmorelli25, just back from PTO.

I agree with @BenB196 that we should add the stack version, to ensure easy tracking of newly added integration as part of a stack release. Integrations can now be shipped out-of-band, but we could still easily tie an integration to a stack release.

A subsequent effort would involve a a changelog for each integration, e.g. if we include bug fixes as part of an integration update - a user should be able to easily identify which bugs were addressed or enhancements made as part of an integration update.

[BenB196]

I wanted to follow up on this issue, as it seems like with 8.x the previous solution doesn't entirely work, especially now that integrations can be released/updated out of normal Elastic release cycles, and it also doesn't appear to be 100% accurate anymore.

Examples of it not being 100% accurate:

If you check 8.2.2 packages, and look for a Mimecast integration

curl "https://epr.elastic.co/search?packageName=aws&kibana.version=8.2.2" | grep -i mimecast

There is nothing there.

But if you go to Kibana (8.2.2 as well) Integrations UI, you can see there is a Mimecast 0.0.12 Beta release Integration available:

A slightly more recent example of this, is 8.3.0 added an Apache Spark integration (According to the release blog: https://www.elastic.co/blog/whats-new-elastic-observability-8-3-0), but if you look at the 8.3.0 output there is nothing that matches spark:

curl "https://epr.elastic.co/search?packageName=aws&kibana.version=8.3.0" | grep -i spark

With the number of Elastic Agent Integrations out there now (looks like ~226 according to the Kibana UI), unless the Integration is specifically called out in something like a release blog, it is becoming a challenge to find newly released integrations.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!