EPR returns almost no packages when kibana.version of 9.0.0 is provided

[kpollich]

When the Kibana version if 9.0.0, almost no packages are returned by EPR, e.g.

https://epr.elastic.co/search?kibana.version=9.0.0

This request only returns the custom logs integration. Adding &prerelease=true makes more packages show up, but it's not much better.

This is blocking Kibana's setup process for the 9.0.0 branch, and version constraints should be permitting 9.0.0 for the Kibana version and showing packages as normal.

Tasks

• @mrodm - Ensure elastic-package can be used to test integrations against the Elastic stack running on 9.0.0 (blocked by Update version to 9.0.0 kibana#192040 and availability of 9.0 snapshots): Add initial support for Elastic Stack 9.x elastic-package#2102
• @mrodm - Update integrations CI to support running tests against a 9.0.0 stack version [CI] Add support for running tests with stack 9.0.0 integrations#11138
• @mrodm - Create a PR that bumps all packages' kibana.version constraint to include || ^9.0.0
• @kpollich - Send an email to elastic-packages@ notifying package maintainers of the PR above. Set a TTL after which the PR will be merged and all bumped packages will become visible + installable in 9.0.0 stacks.

[jlind23]

After chatting with Mario, this is working as designed.
Every package will have to update their constraint to something like:

kibana.version: '^8.15.0 || ^9.0.0'

This has to be requested to all packages owners. @kpollich is this something you can take care of? It has to be done before 9.0 public release.

[jlind23]

One quick and dirty fix could be to add a special case in package registry that will check if package.version = 9.0.0 then return only rely on spec.minand spec.max. @mrodm @kpollich wdyt?

[cmacknz]

Can we just open a PR making this change this for every package? Then package owners can tell us explicitly which packages should not have a ^9.0 constraint, if any?

Potentially also there should be a lint rule that warns if you are not compatible with the latest stack version, since except for packages being actively deleted or deprecated that should always be the case.

If I look at the Elastic Agent package for example it is missing an explicit opt in to 9.0. https://github.com/elastic/integrations/blob/8711ef8d8e85c7c9288cc8d8140c68b4d04cad6a/packages/elastic_agent/manifest.yml#L9-L10

[jlind23]

Can we just open a PR making this change this for every package?

You mean automatically create a PR that will add this constraint for all packages?
@mrodm how complex/hard do you think this would be?

[kpollich]

Can we just open a PR making this change this for every package? Then package owners can tell us explicitly which packages should not have a ^9.0 constraint, if any?

I think this is potentially a bit of an overreach on our part, but also I think package maintainers will simply move too slowly unless we do this (see secrets). Coordinating config changes across all integration maintainers has historically been a slow and painful process.

I'm +1 on creating a PR that bumps every package, except for those that are formally deprecated. For deprecated packages we should give the maintainers a chance to confirm whether the packages should continue to be installable in 9.0.

Potentially also there should be a lint rule that warns if you are not compatible with the latest stack version, since except for packages being actively deleted or deprecated that should always be the case.

This would be a good enhancement and should be relatively easy to add. elastic-package should log a warning a when building or testing a package that isn't compatible with the latest major.

[kpollich]

For next steps, I propose the following:

1. @mrodm - Create the PR bumping the constraints as described above, ensuring codeowners are pinged for review
2. @kpollich - Send an email to elastic-packages@ broadcasting the changes we're making on behalf of package maintainers, linking to the PR. Set a TTL on the email of 4 weeks. Once the TTL expires, the PR will be merged.

There is a small potential for packages to regress in 9.0 due to underlying breaking changes in Elasticsearch mappings/settings, Kibana dashboard behavior, etc, but I strongly doubt we'll have any cases of this.

[mrodm]

Can we just open a PR making this change this for every package?

You mean automatically create a PR that will add this constraint for all packages? @mrodm how complex/hard do you think this would be?

I think this is potentially a bit of an overreach on our part, but also I think package maintainers will simply move too slowly unless we do this (see secrets). Coordinating config changes across all integration maintainers has historically been a slow and painful process.

I'm +1 on creating a PR that bumps every package, except for those that are formally deprecated. For deprecated packages we should give the maintainers a chance to confirm whether the packages should continue to be installable in 9.0.

Not sure about this. Developers/teams cannot test the packages they own with a 9.0.0 stack. So they would not know if the package (as it is) it is supported there.

First, it would be needed to prepare at least the CI pipelines of the integrations repository to at least be ready to test with that stack (maybe daily?).

About not to update deprecated packages, AFAIK currently there is no way (in the spec) to mark a package as deprecated. So not really sure, which ones should be considered as deprecated (elastic/package-spec#227).

[kpollich]

Not sure about this. Developers/teams cannot test the packages they own with a 9.0.0 so they would not know if the package (as it is) it is supported there.

First, it would be needed to prepare at least the CI pipelines of the integrations repository to at least be ready to test with that stack (maybe daily?).

Yes, let's make sure we have the ability to test packages against the 9.0 version of the stack both via elastic-package and in integrations CI before we send out the email and create the bumping PR. I'll add a list of tasks to the description here for what needs to be done.

About not to update deprecated packages, AFAIK currently there is no way (in the spec) to mark a package as deprecated. So not really sure, which ones should be considered as deprecated.

You're right - we don't support this today, e.g. elastic/package-spec#227

[jlind23]

Yes, let's make sure we have the ability to test packages against the 9.0 version of the stack both via elastic-package and in integrations CI before we send out the email and create the bumping PR. I'll add a list of tasks to the description here for what needs to be done.

This won't be possible before the next unified release and once this is done anyone testing 9.0 won't have any package so it is probably worth sending out a communication about this caveat and the fact that we are working on a resolution. Thoughts?

[kpollich]

This won't be possible before the next unified release and once this is done anyone testing 9.0 won't have any package so it is probably worth sending out a communication about this caveat and the fact that we are working on a resolution. Thoughts?

I think this is okay from the integration maintainer standpoint. What we need to be able to do is run elastic-package test against a 9.0.0 snapshot build of the Elastic stack. It doesn't matter if packages don't appear in Kibana for that stack because we're just testing a single integration against Kibana + ES on 9.0.0. We also need to make sure that any packages that have already been bumped to 9.0.0 can run against that stack version in integrations CI.

[mrodm]

@mrodm - Ensure elastic-package can be used to test integrations against the Elastic stack running on 9.0.0 (blocked by elastic/kibana#192040 and availability of 9.0 snapshots)

Related to this item, what would be the elastic-agent docker image to be released in 9.0? Maybe this also requires changes in elastic-package to set the right docker images.

As it is now, elastic-package will try to run in the stack docker.elastic.co/elastic-agent/elastic-agent-wolfi:<version>. Should it be kept for now this docker image (with wolfi prefix)? would that one be available for 9.0.0 ?

[jlind23]

In 9.0 wolfi will became the default and ubuntu images will be removed so it will be rolled back to docker.elastic.co/elastic-agent/elastic-agent:<version>

[jsoriano]

Create a PR that bumps all packages' kibana.version constraint to include || ^9.0.0

We have to be cautious with this. If we release GA packages with 9.x support now, and a breaking change that affects packages is merged in the stack before the feature freeze, we will have broken packages.

So I recommend holding on that till the feature freeze or a close date. In the meantime we can use other approaches like disabling this constraint in kibana 9.0 snapshots.

[kpollich]

We have to be cautious with this. If we release GA packages with 9.x support now, and a breaking change that affects packages is merged in the stack before the feature freeze, we will have broken packages.

So I recommend holding on that till the feature freeze or a close date. In the meantime we can use other approaches like disabling this constraint in kibana 9.0 snapshots.

This makes sense to me. It sounds like we should simply remove the version gating from main in Kibana as 9.0 is quite far out still, then work on getting these packages bumped over a longer period of time.

[kpollich]

@jsoriano - I think we are in an okay spot with this now, but we will need to think more about long-term fixes for 9.0 here as discussed. I'm assigning this to you. Could you provide a summary of where we are today and what needs to be done next at this point? I think reaching out to package maintainers to start adding conditions for 9.0 to packages is next up, but not sure if we need to discuss further.

[jsoriano]

I think reaching out to package maintainers to start adding conditions for 9.0 to packages is next up, but not sure if we need to discuss further.

We shouldn't add 9.0 conditions till we have more certainty on breaking changes in the stack that might affect packages. I would wait till some weeks before the FF to start adding conditions. I would think on a plan like the following one:

• [Done] Remove kibana constraint checks.
• [From now till ~6 weeks before 9.0 FF] Assess breaking changes, early tests on 9.0 code. Keep an eye on daily builds.
• [4 weeks before 9.0 FF] Development teams start adapting constraints, to a small selection of packages, at this point teams should have an idea of packages that they leave behind, if any.
• [2 weeks before 9.0 FF] General adaption of constraints, at least done for most used packages.
• [~1 week before 9.0 FF], enable kibana constraint checks.
• [During FF] There is time to continue adapting constraints.

We could also think on continuing with the plan to remove kibana version constraints, and rely only on spec versions and capabilities, as we already do in serverless. This plan should take into account that we still have to support versions of the stack that don't know anything about spec constraints, but maybe we could handle their behaviour from the registry. If we go this way, then the plan becomes:

• We won't enable kibana version constraint checks in kibana 9.0.
• We ensure that spec [and capability] constraints are used in all 9.0 offerings.
• Package Registry is modified to don't return packages without kibana.version constraint to search queries that include kibana.version. [For compatibility with old versions of kibana].
• Packages that don't need to support versions older than 9.0 can remove their kibana.version constraint. [This would be the only task that requires an action from package developers].
• We consider updating spec.min constraint in kibana.
• With this we lose the opportunity to stop supporting packages, but unmaintained packages will be eventually let out by updates of spec.min constraint.

[jsoriano]

I think this issue can be closed as the issue was found and addressed. We have internal issues to keep track of the pending changes in adaptation for 9.0. Please reopen if something else is needed now.