Upgrade default CNB builders to Paketo Jammy

[scottfrederick]

The default CNB builders used by the Maven and Gradle plugins should be upgraded from the Ubuntu 18.04 based paketobuildpacks/builder:base and paketobuildpacks/builder:tiny to the Ubuntu 22.04 based paketobuildpacks/builder-jammy-base and paketobuildpacks/builder-jammy-tiny.

[wilkinsona]

Re-opening as this seems to be causing a hang on CI when pulling docker.io/paketobuildpacks/builder-jammy-base. The test that hangs is BuildImageTests.failsWhenBuildImageIsInvokedOnMultiModuleProjectWithBuildImageGoal. I used fly hijack to gather some diagnostics.

Log from the Maven build:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] build-image-multi-module                                           [pom]
[INFO] library                                                            [jar]
[INFO] app                                                                [jar]
[INFO] Downloading from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewo
rk/boot/spring-boot-maven-plugin/3.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewor
k/boot/spring-boot-maven-plugin/3.2.0-SNAPSHOT/maven-metadata.xml (1.4 kB at 44 kB/s)
[INFO] 
[INFO] -----< org.springframework.boot.maven.it:build-image-multi-module >-----
[INFO] Building build-image-multi-module 0.0.1.BUILD-SNAPSHOT             [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] >>> spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) > package @ build-image-multi-module >>>
[INFO] 
[INFO] <<< spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) < package @ build-image-multi-module <<<
[INFO] 
[INFO] 
[INFO] --- spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) @ build-image-multi-module ---
[INFO] Downloading from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewo
rk/boot/spring-boot-buildpack-platform/3.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewor
k/boot/spring-boot-buildpack-platform/3.2.0-SNAPSHOT/maven-metadata.xml (1.4 kB at 469 kB/s)
[INFO] Downloading from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewo
rk/boot/spring-boot-loader-tools/3.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from local.central: file:/tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin/build/int-test-maven-repository/org/springframewor
k/boot/spring-boot-loader-tools/3.2.0-SNAPSHOT/maven-metadata.xml (1.4 kB at 467 kB/s)
[INFO] 
[INFO] --< org.springframework.boot.maven.it:build-image-multi-module-library >--
[INFO] Building library 0.0.1.BUILD-SNAPSHOT                              [2/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] >>> spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) > package @ build-image-multi-module-library >>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ build-image-multi-module-library ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /tmp/maven-build10805557081838015816/library/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ build-image-multi-module-library ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /tmp/maven-build10805557081838015816/library/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ build-image-multi-module-library ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /tmp/maven-build10805557081838015816/library/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ build-image-multi-module-library ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ build-image-multi-module-library ---
[INFO] No tests to run.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ build-image-multi-module-library ---
[INFO] Building jar: /tmp/maven-build10805557081838015816/library/target/build-image-multi-module-library-0.0.1.BUILD-SNAPSHOT.jar
[INFO] 
[INFO] <<< spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) < package @ build-image-multi-module-library <<<
[INFO] 
[INFO] 
[INFO] --- spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) @ build-image-multi-module-library ---
[INFO] Building image 'docker.io/library/build-image-multi-module-library:0.0.1.BUILD-SNAPSHOT'
[INFO] 
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 0%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 2%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 5%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 9%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 12%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 15%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 16%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 31%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 43%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 48%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 53%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 59%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 65%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 69%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 75%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 79%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 66%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 67%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 67%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 69%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 70%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 70%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 72%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 72%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 73%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 76%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 76%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 77%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 77%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 79%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base' 82%

Stack trace of the main thread in the Maven build:

"main" #1 prio=5 os_prio=0 cpu=4536.10ms elapsed=367.07s tid=0x00007f813c026150 nid=0x9b87 runnable  [0x00007f81409fc000]
   java.lang.Thread.State: RUNNABLE
	at org.springframework.boot.buildpack.platform.socket.DomainSocket.read(Native Method)
	at org.springframework.boot.buildpack.platform.socket.DomainSocket.read(DomainSocket.java:82)
	at org.springframework.boot.buildpack.platform.socket.DomainSocket$DomainSocketInputStream.read(DomainSocket.java:168)
	at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
	at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:210)
	at org.apache.hc.core5.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:319)
	at org.apache.hc.core5.io.Closer.close(Closer.java:48)
	at org.apache.hc.core5.http.impl.io.IncomingHttpEntity.close(IncomingHttpEntity.java:112)
	at org.apache.hc.core5.http.io.entity.HttpEntityWrapper.close(HttpEntityWrapper.java:120)
	at org.apache.hc.client5.http.impl.classic.ResponseEntityProxy.close(ResponseEntityProxy.java:180)
	at org.apache.hc.core5.io.Closer.close(Closer.java:48)
	at org.apache.hc.core5.http.message.BasicClassicHttpResponse.close(BasicClassicHttpResponse.java:93)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpResponse.close(CloseableHttpResponse.java:200)
	at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport$HttpClientResponse.close(HttpClientTransport.java:262)
	at org.springframework.boot.buildpack.platform.docker.DockerApi$ImageApi.pull(DockerApi.java:198)
	at org.springframework.boot.buildpack.platform.build.Builder$ImageFetcher.pullImage(Builder.java:235)
	at org.springframework.boot.buildpack.platform.build.Builder$ImageFetcher.fetchImage(Builder.java:226)
	at org.springframework.boot.buildpack.platform.build.Builder.build(Builder.java:104)
	at org.springframework.boot.maven.BuildImageMojo.buildImage(BuildImageMojo.java:240)
	at org.springframework.boot.maven.BuildImageMojo.execute(BuildImageMojo.java:230)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:77)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke([email protected]/Method.java:568)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

The response is being closed which causes ChunkedInputStream to read the remainder of the chunked message. The resulting read never completes and never times out so the test never ends.

I am going to revert the change for now to see if we can get CI working again.

[wilkinsona]

One more datapoint: it appears that it's only failsWhenBuildImageIsInvokedOnMultiModuleProjectWithBuildImageGoal that is affected. Killing the Maven process for each of the three runs of that test (Maven 3.6.3, Maven 3.8.5, Maven 3.9.0) allows the build to continue and all of the other tests in BuildImageTests then succeed. Here is a build scan from the build run in the hijacked container where I manually killed the Maven Launcher process for each run of failsWhenBuildImageIsInvokedOnMultiModuleProjectWithBuildImageGoal.