Container will not start when deployed to OpenShift 3.11 with chmod /dev/stdout /dev/stderr permission error

[ToniCipriani]

Description:

When deploying the Envoy container to OpenShift 3.11, the pod goes into a crash loop when starting with a permission denied error. Pod is trying to chmod on /dev/stderr and /dev/stdout. Issue seems to be due to the chmod command in the docker-entrypoint.sh.

Repro steps:

Deploy demo config to OpenShift

Admin and Stats Output:

N/A
Config:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout

Logs:

chown: /dev/stdout: Permission denied
  chown: /dev/stderr: Permission denied

[phlax]

/assign phlax

[phlax]

hi @ToniCipriani can you tell me the user you are starting the container as

[ToniCipriani]

On OpenShift, it runs as a randomly generated user ID. I haven't specified ENVOY_UID or GID.

/ $ ps aux
PID   USER     TIME  COMMAND
    1 10001100  0:32 envoy -c /etc/envoy/envoy.yaml
  126 10001100  0:00 /bin/sh
  134 10001100  0:00 /bin/sh
  146 10001100  0:00 ps aux
/ $ whoami
whoami: unknown uid 1000110000
/ $ 

[ToniCipriani]

I also tried downgrading to the 1.15 and 1.14 containers, and managed to get 1.14.5 working without the error. However I don't think this is ideal.

[phlax]

it needs to run as root - the entrypoint script drops permission to the envoy user

[phlax]

see here for more info https://www.envoyproxy.io/docs/envoy/latest/start/docker#permissions-for-running-docker-envoy-containers-as-a-non-root-user

[ToniCipriani]

Running as root is not an option here, this will be flagged by the company's security. I do see in the gateway-proxy of Gloo (which I believe is based on Envoy) has an option of using the container as a floating user? I think that is the issue here. Is it possible to configure Envoy to run as such?

Gloo provides support for running the gateway-proxy (i.e. Envoy) as an unprivileged container and without needing the NET_BIND_SERVICE capability (note that this means the proxy can not bind to ports below 1024).

https://docs.solo.io/gloo-edge/latest/installation/platform_configuration/cluster_setup/#openshift

And AFAIK it is not necessary to run the chown command on /dev/stdout and /dev/stderr for it to write properly, I'm not aware of any other container that does this.

[ToniCipriani]

I also tried taking your advice to set ENVOY_UID/GID as 0, now the container starts but no logs are observed in /dev/stdout. Ideally we want these logs to be there to be picked up by ELK.

[phlax]

hi - so to clarify envoy/user id

Envoy has an entrypoint that drops permissions from root to envoy inside the container.

This means to use this container it must be started as root - ie non-optional

If you do not want to start the container as root you will need to change the entrypoint - that would work fine - you should be able to run as the envoy uid (101) and things will mostly work. you may have some issue with stdout/err still

[phlax]

and to be clear - im not suggesting setting ENVOY_UID which would actually make the container run as root (edit: if set to 0)

[phlax]

@ToniCipriani see #14141

[phlax]

@ToniCipriani there is an underlying problem here i think in that if you start the container as non-root it doesnt work and doesnt notify you why not

Its not an uncommon pattern to drop root in the entrypoint (otherwise gosu and suexec would not exist)

It is also possible to run this container (with some big caveats/limitations) as non-root without the entrypoint - but its not currently set up to do that (which is why i opened #14141 )

In the meantime, while that issue is considered/discussed/resolved, if you wish to run the container as non-root you will have to hack the entrypoint yourself.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.