Restrospective 2.1.0 release

[gaiksaya]

How to use this issue?

Please add comments to this issue, they can be small or large in scope. Honest feedback is important to improve our processes, suggestions are also welcomed but not required.
What will happen to this issue post release?

There will be a discussion(s) about how the release went and how the next release can be improved. Then this ticket will be updated with the notes of that discussion along side action items.

[gaiksaya]

Problem :
Due to large value of cache invalidation time for min snapshots, the URL returned an old artifact causing the plugin teams's CI to fail.

Action Item :
We need to lower the cache invalidation time for /snapshots/* in order to resolve this.

[amitgalitz]

Not sure if this is the best place for this as its not an exact action item as it might be combination of things that plugins, core and infra could be doing better, basically a lot of small issues are only being discovered the day before the code complete day.
Too break this down to two separate problems:

1. Plugins are finding out last second that they have to make changes before a release code complete deadline
2. There are breaking changes from core on minor version or at least changes that require a fix on plugin side that are being made on minor version. There should be an easy way to inform plugins if such changes will be happening or to stop them from happening.

For problem 1, I do believe that some of it can be solved with a version increment to 2.1 for plugins exactly when core increments version. However in the past this hasn’t been possible because different things like docker isn’t ready yet, some plugins are dependent on others (JS, common-utils) to be incremented first and etc.

• For example AD tried to increment versions to 1.3.1 to 1.3.2 at the time and had a failing build for 20 days because docker wasn’t ready yet for 1.3.2 and a part of our CI depends on it. To fix this we had to comment out the docker related part in our CI and uncomment them later.

For problem 2, it makes sense that sometimes errors are made and things get pushed that lead to issues in compatibility with plugins, this is something that can’t be stopped but the point of using a -SNAPSHOT release is so we find this out as soon as possible which leads back to needing to solve problem 1.
For example a few plugins had to end up making changes last second to accommodate for a new Lucene version that broke several plugins CIs (I believe this was found out earlier by some plugins but maybe should have led to a campaign for all plugins to check), and also changing methods like fromMediaTypeOrFormat to fromMediaType due to this change in core (https://github.com/opensearch-project/OpenSearch/pull/3077/files)

[gaiksaya]

Delay in generating release notes lead to overall delay in release. Release notes need to created at the same time as of code complete date. Additional PRs with bug fixes during the release testing phase can go in later.

[bbarani]

Automating the release notes is tracked here

[bbarani]

Closing this issue as we have successfully release 2.1.0 version