Cannot run image running with container generated by jib as non-root in k8s

[JamesWojewoda]

Description of the issue: I cannot run the image generated via jib-maven-plugin as nonroot. When attempting to set the securitycontext runasuser as a custom one, it errors as the app cannot find the main class. Only root seems to work, but this is a security issue and I'd rather run the container as a unpriviledged user.

Expected behavior: set the security context in the k8s deployment as uid 10000 (just an example), and the app thats generated has /app folders owned or at least executable by this uid.

Steps to reproduce: create a k8s deployment, set securitycontext: runasUser: to custom uid. use a custom jdk image and use that USER uid in the from block of the maven-jib-plugin section in the pom.xml

Environment:

jdk image utilized by pom to generate the new image for the java app:

FROM openjdk:8-jdk-alpine

RUN set -x && \
	sed -i -e "s/\#networkaddress.cache.ttl=-1/networkaddress.cache.ttl=300/g" -e "s/securerandom.source=file\:\/dev\/random/securerandom.source=file\:\/dev\/\.\/urandom/g" /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/java.security

ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"

RUN addgroup -g 10000 runner && \
    adduser -G runner -D -H -u 10000 runner

USER 10000

jib-maven-plugin Configuration:

<plugin>
	        <groupId>com.google.cloud.tools</groupId>
	        <artifactId>jib-maven-plugin</artifactId>
	        <version>0.9.7</version>
	        <configuration>
	          <from>
	          	<image>XXXXXXX.dkr.ecr.us-east-1.amazonaws.com/foo:custom-jdk</image>
	          </from>
	          <to>
	            <image>XXXXXXX.dkr.ecr.us-east-1.amazonaws.com/foo:java-app</image>
	            <credHelper>ecr-login</credHelper>
	          </to>
	          <container>
	          	<ports>
	          		<port>8080</port>
	          	</ports>
	          </container>
	          <imageFormat>Docker</imageFormat>
	        </configuration>
	      </plugin>

Log output: Error: Could not find or load main class com.test.hello.HelloApplication

Additional Information: the folders that are generated in /app are owned by root. resources is available to all users, but the other 2 are root write and executable only.

k8s deployment container:

- name: hello-app
        imagePullPolicy: Always
        securityContext:
          runAsUser: 10000
          allowPrivilegeEscalation: false
        image: XXXXXXX.dkr.ecr.us-east-1.amazonaws.com/foo:java-app

[TadCordle]

Hi @JamesWojewoda, I believe this is related to #523. I think we have a fix planned for not this upcoming release, but the release after (0.9.9).

[briandealwis]

@JamesWojewoda your issue sounds similar to this question:

https://stackoverflow.com/questions/46769339/how-to-change-permission-of-mapped-volume-in-kubernetes-docker

[JamesWojewoda]

@TadCordle Yeah, I think you're on the money. That's good to know! Looking forward to it cause using jib so far has been great.

@briandealwis I'll try tinkering with the volume. Currently I can't seem to mount anywhere but the /app/resources folder.

[coollog]

Another option may be to add configuration for the user (uid, groupid, names..) and write the files with the configured user and set the container config User field to be that as well (so the process runs with that user).

[coollog]

Hi @JamesWojewoda I filed #772 with a potential fix. Would you mind trying that fix to see if it works for your error?

1. Clone the jib repo: git clone https://github.com/GoogleContainerTools/jib
2. Check out the branch git checkout add-extraction-path-to-tar
3. Install the SNAPSHOT version: cd jib/jib-maven-plugin && ./mvnw -DskipTests install
4. Use the 0.9.9-SNAPSHOT version in your project.

[JamesWojewoda]

@coollog Installed and tested, I see that the subdirs of /app are now readable/executable by other users. Still getting the error however, is this possibly due to /app still having only root access?

[coollog]

Hi @JamesWojewoda , I just pushed a commit to add /app as well - can you try this again?

[JamesWojewoda]

@coollog After install, I'm seeing the other root dirs as readable and executable, however /app is still only root accessible

# ls -ltr
total 56
....
drw-------    5 root     root          4096 Aug  2 14:37 app

[coollog]

@JamesWojewoda Oh, maybe try running your Maven build with -U: ./mvnw -U clean compile jib:build. This will force Maven to update snapshot versions used.

[JamesWojewoda]

@coollog Yup that was it, thanks for that. I thought I deleted the locally held jib and upped the version, possibly something wonky happened with my iterm and using multiple windows.

[coollog]

@JamesWojewoda Okay great! Glad that worked - we will have that fix in for version 0.9.9.

[JamesWojewoda]

@coollog The app dir has been updated, unfortunately I'm still seeing the error cant find main class - but I did notice the com subdir getting created under /app/classes is still only available to root.
/app # ls -ltr
total 20
drwxr-xr-x 3 root root 4096 Jan 1 1970 classes
drwxrwxrwx 3 root root 4096 Aug 2 19:28 resources
drwxr-xr-x 2 root root 12288 Aug 2 19:28 libs
/app # ls -ltr classes
total 4
drw------- 4 root root 4096 Aug 2 19:28 com
/app # ls -ltr classes/com/
total 8
drwxr-xr-x 4 root root 4096 Jan 1 1970 tnsi

[chanseokoh]

@coollog Unrelated, but I'm curious why resources folder has a different permission set (777).

[coollog]

@chanseokoh Yea, this is definitely quite puzzling.
@JamesWojewoda Can you check what the permissions for your target/classes directory folders are? (and specifically what target/classes/com's permissions are)

[JamesWojewoda]

@coollog Heres the permissions on my jenkins maven container that uses jib to build the image:

/target/classes
drwxr-xr-x 3 10000 root 4096 Aug 3 13:31 classes

/target/classes/com
drwxr-xr-x 4 10000 root 4096 Aug 3 13:31 com

/target/classes/com/(classes in the proj)
drwxr-xr-x 4 10000 root 4096 Aug 3 13:31 tnsi

[coollog]

Hi @JamesWojewoda , can you try again with the latest change? I pushed a fix to set permissions for all directories.

1. Clone the jib repo: git clone https://github.com/GoogleContainerTools/jib
2. Check out the branch git checkout add-extraction-path-to-tar
3. Install the SNAPSHOT version: cd jib/jib-maven-plugin && ./mvnw -U -DskipTests clean install
4. Use the 0.9.9-SNAPSHOT version in your project.

[JamesWojewoda]

@coollog When using an image generated via that branch, I'm getting the following error:

rpc error: code = Unknown desc = failed to register layer: ApplyLayer duplicates of file paths not supported stdout: {"layerSize":349759}

The image was put into an ECS repo, and when trying to pull and run the image locally I get:
failed to register layer: Error processing tar file(duplicates of file paths not supported)

[coollog]

Hi @JamesWojewoda , I made some changes to guarantee no duplicates in the tarball entries. Can you try again and see if it works now?

1. Check out the branch git checkout add-extraction-path-to-tar
2. Pull latest changes git pull
3. Install the SNAPSHOT version: ./mvnw -U -DskipTests clean install
4. Use the 0.9.9-SNAPSHOT version in your project.

[JamesWojewoda]

@coollog Now working, however the dirs have gone back to being root only.

container image:
/
drw------- 5 root root 4096 Aug 8 18:58 app

/app
drwxrwxrwx 3 root root 4096 Aug 8 18:57 resources
drw------- 2 root root 12288 Aug 8 18:58 libs
drw------- 3 root root 4096 Aug 8 18:58 classes

/app/classes
drw------- 4 root root 4096 Aug 8 18:58 com

/app/classes/com
drwxr-xr-x 4 root root 4096 Jan 1 1970 tnsi

target folder:
/target
drwxr-xr-x 4 jwojewoda staff 136 Aug 8 14:55 classes
/target/classes
drwxr-xr-x 4 jwojewoda staff 136 Aug 8 14:55 com
/target/classes/com
drwxr-xr-x 4 jwojewoda staff 136 Aug 8 14:55 tnsi

[coollog]

@JamesWojewoda Hmm, this is a puzzler. Could you try building to a Docker daemon with (mvn compile jib:dockerBuild) and then run that 1) with docker locally and 2) also push that image to your registry and run on Kubernetes?

[JamesWojewoda]

@coollog It failed trying to build it with the docker daemon as well.

Caused by: java.io.IOException: 'docker load' command failed with output: at com.google.cloud.tools.jib.docker.DockerClient.load (DockerClient.java:110) at com.google.cloud.tools.jib.builder.steps.LoadDockerStep.afterPushBaseImageLayerFuturesFuture (LoadDockerStep.java:92) at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly (CombinedFuture.java:181) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:57) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641) at java.lang.Thread.run (Thread.java:844)

[coollog]

@JamesWojewoda Hmm, could you try building the tarball and doing docker load manually? Like:

mvn compile jib:buildTar
docker load --input target/jib-image.tar

[JamesWojewoda]

@coollog Same issue, I'm afraid. The tar builds ok however.

Error processing tar file(duplicates of file paths not supported):

[paivagustavo]

I'm also having the same issue, if needs some help to debug just get in touch with me too.

[coollog]

Hmm, this looks like the same issue happening at bazelbuild/rules_docker#246

[coollog]

@JamesWojewoda the original fix had a bug that caused the Error processing tar file error. I filed a new fix at #891. The error should go away with that now.

After some more discussions, we are planning to add a configuration to jib-maven-plugin/jib-gradle-plugin that can define the userid/groupid to set for the files we build into the container, similar to how Dockerfile's COPY --chown works. So, for example, in your case, the configuration could look something like:

<configuration>
  ...
  <container>
    <chown>10000</chown>
  </container>
</configuration>

This will make sure all the files are created with 10000 as the uid.

[coollog]

@JamesWojewoda Can you try again with the fix at #891? It might just work for your case and wouldn't need the chown configuration.

1. Check out the branch git checkout add-extraction-path-to-tar
2. Pull latest changes git pull
3. Install the SNAPSHOT version: ./mvnw -U -DskipTests clean install
4. Use the 0.9.10-SNAPSHOT version in your project.

[JamesWojewoda]

@coollog This works great. Thanks for your help!

[paivagustavo]

@coollog I had the same problem as @JamesWojewoda and can confirm that after the recent changes, the image was successfully loaded and the permissions was right as you can see:

$ ls -lah app
total 16K
drwxr-xr-x. 5 root root 50 Jan 1 1970 .
drwxr-xr-x. 1 root root 57 Aug 29 13:58 ..
drwxr-xr-x. 3 root root 16 Jan 1 1970 classes
drwxr-xr-x. 2 root root 12K Aug 29 13:58 libs
drwxr-xr-x. 3 root root 50 Aug 29 13:58 resources

Much appreciated those changes 👍

[coollog]

@JamesWojewoda Awesome! We'll get that merged in for 0.9.10 soon :)

[coollog]

@GuustavoPaiva Great!

[coollog]

@JamesWojewoda @GuustavoPaiva We have released version 0.9.10!

[yangguang001]

mvn compile jib:buildTar
docker load --input target/jib-image.tar
error like this:
unsupported os linux

[briandealwis]

@yangguang001 please open a new issue and include:

1. the build output that you see
2. the result of file target/jib-image.tar
3. the result of tar tvf target/jib-image.tar
   Thanks!