exec: "sleep 86400": executable file not found in $PATH

[sosiouxme]

I must be doing something wrong here but can't figure out what.

With imagebuilder from latest head, and docker-1.13.1-51.git4032bd5.fc27.x86_64

$ imagebuilder -t foo -f Dockerfile.min .
--> FROM fedora:27 as 0
--> RUN touch /etc/foo
--> Committing changes to foo ...
--> Done
$ docker run -it --rm foo 
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"sleep 86400\": executable file not found in $PATH".
$ docker run -it --rm foo /bin/bash
[root@7e2cdeabd622 /]# exit
$ docker run -it --rm foo sleep 1
$

First of all, where did "sleep 86400" come from? fedora:27 has no Cmd or Entrypoint set. foo does.

$ docker run -it --rm fedora:27
/usr/bin/docker-current: Error response from daemon: No command specified.
See '/usr/bin/docker-current run --help'.
$ docker inspect foo
[
    {
        "Id": "sha256:948b2333e11c0dfbedf4bde7e7a7fe6749376dd5d5fae112992edfa600838ad8",
        "RepoTags": [
            "foo:latest"
        ],
        "RepoDigests": [],
        "Parent": "sha256:9110ae7f579f35ee0c3938696f23fe0f5fbe641738ea52eb83c2df7e9995fa17",
        "Comment": "",
        "Created": "2018-04-18T19:01:37.421678496Z",
        "Container": "fc1eb310d84b08683a2eab434268e46313bdf46b2a533606e4cdbed1a5e7727e",
        "ContainerConfig": {
            "Hostname": "fc1eb310d84b",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f27container",
                "FGC=f27",
                "FBR=f27"
            ],
            "Cmd": [
                "sleep 86400"
            ],
            "Image": "fedora:27",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/bin/sh",
                "-c"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "1.13.1",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f27container",
                "FGC=f27",
                "FBR=f27"
            ],
            "Cmd": [
                "sleep 86400"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [],
            "OnBuild": null,
            "Labels": {}
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 235250693,
        "VirtualSize": 235250693,
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/70bea816b920919e3afcdaca1f7c572c83308f4bee6379450e17acba57da28ff/diff",
                "MergedDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/merged",
                "UpperDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/diff",
                "WorkDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:39bae602f7539e626f6894ecded6d12a6c56745dbab9aee940e258c766e090e8",
                "sha256:de41557ebc416b99c4bb90ae5e39f4c451055236db8aff0cbe10f8206a275039"
            ]
        }
    }
]

Second, why does it fail to run that?

[smarterclayton]

The container image builder creates has to have something to hold it open. All RUN commands are from exec.

I did a docker run equivalent and it worked for me. Not sure what the difference is.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sosiouxme]

/reopen
it's still a problem and causing some amount of grief in OSBS builds, FWIW

[openshift-ci]

@sosiouxme: Reopened this issue.

In response to this:

/reopen
it's still a problem and causing some amount of grief in OSBS builds, FWIW

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sosiouxme]

/remove-lifecycle rotten
/lifecycle frozen

[nalind]

imagebuilder keeps track of the command and entrypoint of the base image, and when it encounters a Dockerfile instruction for changing them, it makes note of them. As @smarterclayton mentioned, it has to supply the engine with some command to make the working container idle so that it can use the "exec" engine API call to handle RUN instructions. Then, at commit-time, imagebuilder supplies the engine either with the value it retrieved from the base image, or the updated value it was given in the Dockerfile.

In the example we're seeing, the base image, fedora:27, doesn't supply an entrypoint or command. Because the build includes a RUN instruction, the working container is created and started with a dummy command ("sleep 86400"). At commit-time, imagebuilder supplies an image configuration structure to the engine where, because the base image didn't supply a value, and the Dockerfile didn't either, the command and entrypoint fields remain blank. The engine starts with the container's configuration, overwrites fields in it with fields that are set in the the image configuration it got from imagebuilder, and writes the image. The method for merging the configuration the client supplies at commit-time with the configuration of the container means that certain fields like the command, entrypoint, and user can't be unset or cleared. They can only be changed to some other non-empty value.

The combination of not having a command in the base image and having RUN instructions is what's triggering the problem, but I don't see any way to clear the field at commit-time in the engine API.

Possible workarounds would be to either not use RUN instructions (imagebuilder doesn't set a command unless it needs to start the working container, and it doesn't need to start a container unless it needs to use an exec call to run a command in it), make sure the base image specifies a command (apparently fedora did starting with fedora:28), or always specify a command in a Dockerfile, even if it's just a CMD ["/bin/bash"].

There's probably an argument to be made that imagebuilder should be supplying a default command and entrypoint when the base image doesn't include one, but I haven't worked out how it would impact tests.