Support building Che on linux on Z (s390x) and Power (ppc64le)

[nickboldt]

Epic issue to collect all subtasks / needs for empowering Che 7.x to build on s390x and ppc64le.

Please add issues below.

• [Enhancement] Extending Eclipse Che CI infrastructure support to s390x [Enhancement] Extending Eclipse Che CI infrastructure support to s390x #16287
• Ensure Che image publishing is multiarch Ensure Che image publishing is multiarch #17124
• plugin registry support for s390x #428
• devfile registry suport for s390x #214
• feat(alpine): Change parent image to be alpine based as well #226
• ...

[shahidhs-ibm]

@nickboldt In order to extend support to s390x, we will need to find suitable replacement for openjdk:8u242-jre-slim docker image located at https://github.com/eclipse/che/blob/master/dockerfiles/che/Dockerfile#L8
Following are some images which support multi-arch and can act as replacement:

• adoptopenjdk:8u232-b09-jre-hotspot
• registry.redhat.io/openjdk/openjdk-8-rhel8:latest
• openjdk:8-jre-alpine

Kindly suggest. We have tested the above images and they work for Intel and s390x architectures. We'll create the PR after confirmation.

[benoitf]

I like openjdk:8-jre-alpine because it is really multi-arch aware
But I know @skabashnyuk is working on moving to java11 so we should switch to something working for java11 as well.

[shahidhs-ibm]

From the online resources what I learned is that OpenJDK11 on Alpine is not officially supported and there are no plans for same in future.
Ref. docker-library/openjdk#403 (comment)
docker-library/openjdk#211 (comment)

@benoitf @nickboldt WDYT? Should we keep support for OpenJDK8 till this activity is complete?

[shahidhs-ibm]

@nickboldt your input? Shall we proceed with creating the PR as per the comment from Florent (openjdk:8-jre-alpine image)?

[skabashnyuk]

@shahidhs-ibm what you can suggest as java11 alternative for Z (s390x) and Power (ppc64le) ?

[shahidhs-ibm]

@skabashnyuk We found openjdk:11.0.3-jre-slim which can serve the purpose. Also this image is Debian based which is similar to one which we are currently using for OpenJDK8. We have tested this image on s390x and have no issues so far. Also this image is multi-arch so should work for ppc64le as well.

@benoitf @nickboldt We're bit confused now on PR to raise. Should we create one with openjdk:11.0.3-jre-slim or with openjdk:8-jre-alpine as per the earlier discussion? We believe that answer to this will be based on your preferences, priority and timelines. What will you suggest?

[nickboldt]

@shahidhs-ibm To be consistent w/ other Z and P images supported by RH, we might want to consider using openj9 instead of openjdk images.

[skabashnyuk]

we might want to consider using openj9 instead of openjdk images

@nickboldt can we start thinking about that after 7.15 and Java 11 migration?

[ghatwala]

Agree @nickboldt - seems openj9 is preferred JVM for Z/P - https://projects.eclipse.org/projects/technology.openj9 .

[ghatwala]

@nickboldt - for openj9 . Were u referring to adoptopenjdk's - OpenJDK + Eclipse OpenJ9 binaries built by AdoptOpenJDK images present at - https://hub.docker.com/_/adoptopenjdk to be used here ?

[prabhav-thali]

@nickboldt @benoitf @skabashnyuk
Since PR#16649 was merged, we observed that java11 image used for che-server does not have multi-arch support. Can it be replaced with multiarch support image?

[prabhav-thali]

Since PR#16649 was merged, we observed that java11 image used for che-server does not have multi-arch support. Can it be replaced with multiarch support image?

@nickboldt @benoitf @skabashnyuk any update?

[skabashnyuk]

@skabashnyuk Even we have got same warnings on ppc64le and investigating the root cause.Not getting warnings immediately, after few hours it is showing.Is it similar at you end?

I'm sorry I don't understand your message. Did you able to find the root cause?

[bivasda1]

@skabashnyuk Even we have got same warnings on ppc64le and investigating the root cause.Not getting warnings immediately, after few hours it is showing.Is it similar at you end?

I'm sorry I don't understand your message. Did you able to find the root cause?

@skabashnyuk Still we are trying to find out the root cause.Have u got those warnings immediately after deployment of che-server?

[skabashnyuk]

@skabashnyuk Still we are trying to find out the root cause.Have u got those warnings immediate after deployment of che-server?

yes

[bivasda1]

@skabashnyuk I am not getting warnings now and also able to login to che-server.Please find the below attached che-server logs.
che-server logs.txt.

[skabashnyuk]

You do have this

20-Jul-2020 07:49:36.280 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en
20-Jul-2020 07:49:36.281 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en
20-Jul-2020 07:49:36.281 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en

if you wait a minute or two (5 maximum) do you see something like this?

failed getting JSON response from Kubernetes Client[masterUrl=https://17

[bivasda1]

@skabashnyuk

if you wait a minute or two (5 maximum) do you see something like this?

failed getting JSON response from Kubernetes Client[masterUrl=https://17

@skabashnyuk I have deployed 1 hour back.still not getting this warning,checking above mentioned warnings.

[prabhav-thali]

@skabashnyuk With the warnings I have mentioned in my comment, I don't see it affecting any functionality on dashboard. Also, found a similar issue #11571. It looks like these warnings existed since JDK8 and not caused due to JDK11 upgrade. Also on intel, observed that these warnings occur with current java11 image (openjdk:11-jre-slim) in Dockerfile.

So is it safe to ignore these warnings? Or should we check for any other alternate docker image?

[bivasda1]

You do have this

20-Jul-2020 07:49:36.280 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en
20-Jul-2020 07:49:36.281 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en
20-Jul-2020 07:49:36.281 CONFIG [localhost-startStop-1] sun.util.locale.provider.LocaleServiceProviderPool.config A locale sensitive service provider returned null for a localized objects,  which should not happen.  provider: sun.util.cldr.CLDRCalendarDataProviderImpl@1ca5752f locale: en

if you wait a minute or two (5 maximum) do you see something like this?

failed getting JSON response from Kubernetes Client[masterUrl=https://17

@skabashnyuk Now I am not getting these warnings in ppc64le. Few Waring's are there that should not affect to the functionality of che-server

Steps taken:

• openjdk:11-jre-slim-stretch image uses JAVA_DEBIAN_VERSION=11.0.3+1-1~bpo9+1 as base. Please refer:
  https://github.com/docker-library/openjdk/blob/178c542fbb93a8f8a42e331b73a1214c9d8ba81d/11/jre/Dockerfile
• Replaced JAVA_DEBIAN_VERSION=11.0.3+1-1~ bpo9+1 with upgraded java version as JAVA_DEBIAN_VERSION 11.0.6+10-1~bpo9+1 and rebuilt the image locally.
• Recreated che-server image with local base image created above and ran it on OCP cluster.
  Server logs are attached here for reference. Please have a look.
  che-server logs.txt

[skabashnyuk]

Few Waring's are there that should not affect to the functionality of che-server

I believe we should not consider these warnings as not important. Have you able to identify the cause?

[bivasda1]

Few Waring's are there that should not affect to the functionality of che-server

I believe we should not consider these warnings as not important. Have you able to identify the cause?
@skabashnyuk I have tested with JAVA_DEBIAN_VERSION=11.0.6+1-1~ bpo9+1 as base (https://packages.debian.org/stretch-backports/java/openjdk-11-jre) and I am not getting any warnings with this particular version.
With JAVA_DEBIAN_VERSION=11.0.3+1-1~ bpo9+1 as base (https://packages.debian.org/stretch-backports/java/openjdk-11-jre-dcevm) I am getting those warnings.
I am still trying to figure out, if something got changed in these versions related to the warnings.

[bivasda1]

@skabashnyuk @gerrith3 @prabhav-thali
The openjdk:11-jre-slim-stretch image was built one and half years ago.
It uses openjdk-11-jdk-headless-11.0.3+1-1~bpo9+1.

Dockerfile path of above docker image: https://github.com/docker-library/openjdk/blob/178c542fbb93a8f8a42e331b73a1214c9d8ba81d/11/jre/slim/Dockerfile
When we tried to recreate the image using below Dockerfile, we got an error.

This happened because, java version 11.0.3+1-1~ bpo9+1 is no longer available and has been replaced by version 11.0.6_10-1~ bpo9+1
Replacing the value 11.0.3+1-1~ bpo9+1 with 11.0.6+10-1~ bpo9+1, in the above mentioned Dockerfile, we were able to re-create the openjdk:11-jre-slim-stretch image.
Now, using this freshly created openjdk:11-jre-slim-stretch image (having openjdk-11-jdk- headless-11.0.6+10-1~ bpo9+1), we created che-server image.
When we tested this new che-server image, we are not getting any warnings as observed earlier with older openjdk:11-jre-slim-stretch image.
Attached is the server log.
che-server log-.txt

Hence we feel that if we have openjdk:11-jre-slim-stretch image available based on latest Java package, then we can create a multi-arch che-server image without any warnings.

[prabhav-thali]

@bivasda1 I have tested the changes on s390x. No warnings were observed after the deployment of che-server. Also, observed an additional change to be incorporated in Dockerfile (which seems to be architecture specific) as mentioned below :

- FROM debian:stretch-slim
+ FROM s390x/debian:stretch-slim

@skabashnyuk any update?

[skabashnyuk]

@prabhav-thali I'm sorry I didn't understand the question to me.

[prabhav-thali]

@skabashnyuk any further update on jdk11 base image change?

[skabashnyuk]

@prabhav-thali I have none at this moment. I didn't understand from @bivasda1 explanation the reason of errors with 11-jre-slim-stretch or proposal of alternative image

[skabashnyuk]

@prabhav-thali @bivasda1 can you please check if s390x/openjdk:11-jre-slim is working for you https://hub.docker.com/layers/s390x/openjdk/11-jre-slim/images/sha256-2f0660a19ac1b4c9f47677d3840624d7ffb0aded45fc170f7bd9576f052db579?context=explore. IF yes we could add extra parameter to adjust FROM here https://github.com/eclipse/che/blob/master/dockerfiles/che/Dockerfile#L22

[ghatwala]

@skabashnyuk - we need ppc64le too , so @bivasda1 can you please check https://hub.docker.com/layers/ppc64le/openjdk/11-jre-slim/images/sha256-57f30fee83b9117237ffd893fb23380c4bcacf4a69e0303300182b4b7416aba9?context=explore works ?

[bivasda1]

@ghatwala ok sure

[prabhav-thali]

@skabashnyuk After using s390x/openjdk:11-jre-slim, observing same warnings in browser console and che pod logs as I had mentioned in this comment.

Env details :
platform: minikube 1.7.2 (with k8s v1.16.1)
installer: helm v3.1.1

With the warnings I have mentioned in my comment, I don't see it affecting any functionality on dashboard. Also, found a similar issue #11571. It looks like these warnings existed since JDK8 and not caused due to JDK11 upgrade.

@skabashnyuk Could you please confirm this at your end?

[skabashnyuk]

@prabhav-thali #16655 (comment) that is acceptable. Overall functionality is working?

[prabhav-thali]

@skabashnyuk As currently we don't have language specific images available for s390x, we were not able to test workspace creation. However the rest functionality looks ok to me and can say its working.

[skabashnyuk]

I'm talking about using that image for che-server, not for workspaces.

[prabhav-thali]

@skabashnyuk we tested s390x/openjdk:11-jre-slim image for che-server and warnings were seen in pod logs. However, no functionality loss is observed.

[benoitf]

I've opened #17866 to discuss the usage of adoptopenjdk/openjdk11:jre with multi-arch

[benoitf]

Usage of adoptopenjdk/openjdk11 has been merged so it should fix the multi-arch build of che-server image

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.