runc fails to start: Container with id exists upon running runc

[rajasec]

I was able to run runc successfully..

Later after exit of the container, I could not launch any of my container image ( eventhough I had rootfs and container.json properly placed)

Code: Id already in use
Message: Container with id exists: runc

I've ensure that no runc process was running earlier.
ps -eaf |grep -i runc
root 10077 9934 0 21:53 pts/31 00:00:00 grep --color=auto -i runc

Thanks
Rajasec

[rajasec]

Steps to reproduce the problem
$HOME/go/runc.
This spawned the new container and entered shell..

For testing purpose whether killing runc will kill the container or not, I went and killed runc process
kill -9 < pid of runc >

Post that, I could not launch any container using runc command

[crosbymichael]

If you SIGKILL runc, it is not able to cleanup it's state on disk and thinks that there is already a container running. So you have to manually remove the directory in /var/run/ocf/<container> to start it again.

[rajasec]

Thanks, it worked..

[CameronNemo]

Maybe actual file locks should be used so they are automatically unlocked when the process dies.

[rajasec]

@crosbymichael

is the below functionality need to be implemented part of runc for cleaning up its state. Below was handled by execdriver in docker
func (d *driver) cleanContainer(id string) error {
d.Lock()
delete(d.activeContainers, id)
d.Unlock()
return os.RemoveAll(filepath.Join(d.root, id))
}

[rajasec]

In runc, the destroy function in run.go is not getting called because of SIGKILL.
defer destroy(container)
As signal can not be catched, some mechanism clean up it state in future

Rajasec

[rajasec]

before the runc starts
Do we need to get similar fix for runc "Cleanup libcontainer container state #11507 "

[crosbymichael]

Maybe we need to implement runc kill or runc signal so that it can send a SIGKILL to the container process and still successfully cleanup state after it dies.

[rajasec]

@crosbymichael

To test whether signals reaching inside the container, I wrote small shell script to catch the signals using trap
handler for SIGINT SIGHUP SIGTERM SIGQUIT SIGKILL

Started the container and my container process waited for signal which is expected.
root 4183 2555 0 17:46 pts/1 00:00:00 ./runc
root 4188 4183 0 17:46 pts/20 00:00:00 /bin/sh /bin/shelltrap

From the host, I've sent the signal to my container process from Host.
kill -1 4188
My container process is able to receive this signal ( SIGHUP)

After that I've done kill -9 4183 ( pid of runc)
Looks like the signal is not flown down to container process( child processes). I thought that killing the parent process using SIGKILL will send the signal to Child process ( looks like it forcefully killed the container too)

~Rajasec

[jhjeong-kr]

I know that current golang cannot catch SIGKILL, so nobody can install the handler.
Therefore, I propose an alternative.

Proposal

1. on creation
   check the container path, [/var/run/ocf/
   ....... if not exist
   .................> create a file "task" in [/var/run/ocf/] which contains pid of the container.
   .................> continue on creating steps
   ....... if exist -> read "task" and extract pid, and then check the process is alive
   ................. if not exist -> yield a user message: "sudo rm -rf /var/run/ocf/
   ................. if exist -> yield a user message: "container is already running"
2. on destroying
   delete "task"
   delete path

[CameronNemo]

@jhjeong-kr runc should just delete the directory itself if the PID is dead and continue (maybe with a warning). That is standard practice with PID files.

[jhjeong-kr]

@CameronNemo Yes, right, but let's suppose the moment that runc finds the lock path(/var/run/ocf/). Current runc just says a fatal message which is not user-friendly. Also the message(Id is in use) is not true in case that runc was terminated by SIGKILL. Therefore, we have to distinguish between abnormal termination(such as SIGKILL) and "in use" if runc finds the lock path.

[rajasec]

@crosbymichael
As per your previous comment, I've generated pull request last week for runc kill which will send signal to the containers.

[mrunalp]

Closing as kill support was added in #178