General Retrospective for September/October 2023 Releases

[adamfarley]

Summary

A retrospective for all efforts surrounding the titular releases.

All community members are welcome to contribute to the agenda via comments below.

This will be a virtual meeting after the release, with at least a week of notice in the #release Slack channel.

On the day of the meeting we'll review the agenda and add a list of actions at the end.

Invited: Everyone.

Time, Date, and URL

Time: 2pm GMT, 9am EST.
Date: 20th of November
URL: https://meet.google.com/tih-uejz-gpq

Details

Retrospective Owner Tasks (in order):

• Post retro URL in #Release around the start of the new release.
• Copy actions from previous retrospectives into this issue, ignoring anything ticked.
  • Include owner names.
• Wait until most builds are released, with no clear signs of additional respins.
• Announce the retrospective's date + time on #Release a week before.
• Host the retrospective:
  • Tick completed actions from last retrospective/s.
  • Go through the agenda.
  • Create a tickable list of actions with owner names.
• Create a new retrospective issue for the next release.
• Set a calendar reminder so you remember to do step 1 before the next release.
• Close this issue.

[adamfarley]

Actions imported from adoptium/adoptium#249

Andrew Leonard:
jdk-21 mac x64 installers failed with an unknown notarize failure: adoptium/installer#737
So mac x64 had to be re-run, however the unsigned installer artifacts were still stored in the top level pipeline, which could lead to confusion especially if the error had not been seen...
Maybe ensure the artifacts are not always stored in the top-level on this sort of failure?
Although hard to distinguish between just a test job failing, and the installer failing...?

Adam Farley:
The dry-runs for JDK21 were run closer to the release date than anticipated.
Advise that we automate the pre-release dry runs to add confidence in build framework stability prior to a release.
This is especially useful for ea-tag-triggered builds which may not have run for weeks (or more) preceding the release.

Adam Farley:
Should we do a community-wide release champion process overview? There seems to be a lack of awareness surrounding the general shape of the process.
To include:

• The main stages.
• When they are carried out, relative to a release.
• What a release checklist is.
• How it is made, and where we put the link.
  Perhaps this could be a short talk at the start of a community call?

[adamfarley]

Unticked actions from previous retrospectives.

July 2023:

• Stewart: To make an issue for reviewing filename expression in RELEASING.md. General Retrospective for July 2023 Releases adoptium#240 (comment) - Raised General Retrospective for September/October 2023 Releases #4
• Scott: To raise issue for regular installer testing (build installer, test it, upload it, download it, install it, java -version).
• Stewart: To discuss future plans for build scheduling at the next community retrospective (ea builds, testing, etc).
  https://docs.google.com/document/d/1oaU8GA_3F8DDzjeZujY10fiyRS-vKdF5rE0eAhAascY/edit#

March 2023:

• Trial the automatic rerun feature for Jenkins test jobs prior to April release to see if we may want to use it for April. Trial the 'automatic rerun on failure' feature for test jobs ci-jenkins-pipelines#777
  • To roll over into next retro.

[sxa]

Recommend not including the checklist issue in the website banner. It contains a level of detail and process which is not useful to the end user trying to determine what is happening.

[sxa]

The website banner stating "We are creating the October 2023 PSU binaries " is not really an accurate status prior to the tags being created upstream. We should avoid posting this banner until at least one of the tags is in place and we are genuinely in the process of creating the binaries.

[andrew-m-leonard]

jdk8 mac x64 build built with an invalid libfreetype.dylib, which is linked with an arm64 libpng16 library.

[jiekang]

Self-reminder to look at adoptium/infrastructure#2886

The machine is still in the pool and the test still fails there.

[adamfarley]

It'd be useful if we could set a default web page for ci.adoptium.net which attempts to access the login system before declaring that the page does not exist.

So:

If not logged in: Attempts to log in and reload, automatically.
If logged in: Declares that the page does not exist or you don't have access. Provides ci.adoptium.net link.
If failed to log in: Declares "Login failed." and provides ci.adoptium.net link.

[andrew-m-leonard]

We unfortunately published jdk-21 x64 mac with un-signed .pkg due to the EF signing service failure being not noticed.
Need to consider implementing adoptium/temurin-build#3494
and also extend to ensure installers are signed too.

[sxa]

Ensure full documentation for releasing with a floating patch is documented (Ref: Example of JDK11/17 on AIX using this branch)

[andrew-m-leonard]

The signing verification issue also hit Windows where jdk.jpackage JMOD executeables were incorrectly signed due to coding error in signing logic.

Fixed logic PR: adoptium/ci-jenkins-pipelines#831

[andrew-m-leonard]

The signing verification issue also hit Windows where jdk.jpackage JMOD executeables were incorrectly signed due to coding error in signing logic.

This would have been visible in nightlies: http://20.90.182.165/output/test?id=65238bbdb6515300abab7025
I am wondering if we should have a new aim of 100% build & test daily status? as it' so easy to miss these otherwise?

[andrew-m-leonard]

Looks like for Windows aqa-tests a lot of test groups only work on certain nodes, I think we need to get this consistent on all nodes, so it doesn't lead to missing real problems.

[sxa]

@andrew-m-leonard Can you add some details of that to your comment? I'm not aware of regularly having many of those and I do sometimes worry that we're a little bit blasé about accepting those sorts of failures.

[sxa]

To make it easier to have "notable updates" as part of the release announcement blogs, perhaps we should created the release status document ahead of time and add a comment, or section in the main description, where people can add things to call out in the release blog. I'm currently thinking there's something extra to call out for this release but not 100% what it might be. Note that adoptium/temurin-build#3484 will pick up some differences so if a change to the checks in there are required they would be good candidates for calling out in the release notes.

[sxa]

Perhaps need something in the releasing docs to ensure that new platforms are included in the release check. I've added Alpine/aarch64 under adoptium/github-release-scripts#141

[adamfarley]

Problem: "Permission to publish" requests can be forgotten.

One solution: Some form of Slack bot that recognises requests to publish, and notifies the poster once the required +1's have been accumulated. This bot could also supply a link that populates all of the relevant parameters correctly, e.g. "Grinder/parambuild/?SDK_RESOURCE=upstream&etcetc".

Said bot could also remind the poster if the expected binary isn't present in the correct repository within an acceptable time period.

Notes:

• If we do do this, we should make sure it can expanded easily, for future functionality.
• This bot would need to populate parameters in a way that plays well with this issue.
• Could such a bot help close the remaining tasks in this issue?

[tellison]

The task to update support dates is marked as completed, but we have received an issue to update them as they were not done.

[adamfarley]

Actions from this Retrospective:

• Adam: To rebook any other retrospective calls to remove time limit and need to "ask" to join.
• Stewart: Make this change to website banner: General Retrospective for September/October 2023 Releases #4 (comment)
• Adam: Do this change in release checklist. General Retrospective for September/October 2023 Releases #4 (comment) - Relocate website banner step #14
• Jie: To fix this test issue upstream by making test more robust in regards to the "sleep" operation.
• Shelley - To update the release checklist to include "create the release draft for the next release" as a post-build step, to enable us to add "notable updates" as comments over time.

Note: The Monday retrospective ended in the middle of discussing this comment. We discussed topics like:

• Creating a new channel for "permission to publish" requests, so they don't get drowned out.
• Asking people to include confirmations/publish job links in the thread under a request.
• For the release champion to scan for these.
• To have some manner of automation relating to the TCK status and release status tables.

To finish that discussion, and handle any further retrospective topics in the weekly community call.

Updates:

• The "Permission to publish" issue will be discussed here.
• The "Update support dates" issue will be discussed here.

[adamfarley]

Closing this retrospective. Future topics should be added to the one for January's releases.