Changes to config.json are not picked up

[don-code]

Describe the bug
If config.json is modified outside of the Watchtower container with docker login, the changes are not reflected by Watchtower until a restart.

To Reproduce
Steps to reproduce the behavior:

1. At runtime, mount in /root/.docker/config.json with the following contents:

{
	"auths": {
	}
}

Watchtower at this point is running as:

[root@ip-10-0-2-15 vagrant]# docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock -v /root/.docker/config.json:/config.json containrrr/watchtower -i 60 --cleanup --debug

2. docker login to some repository, which updates /root/.docker/config.json.
3. Run a container from the private registry corresponding with the repo from step two. Here I'm using an image in ECR to demonstrate, but this does not appear limited to the remote repository being ECR or not.

[root@ip-10-0-2-15 vagrant]# docker run 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest sleep 100000

4. Observe that Watchtower cannot fetch updates for this container.

time="2020-02-20T17:39:48Z" level=debug msg="Checking containers for updated images"
time="2020-02-20T17:39:48Z" level=debug msg="Retrieving running containers"
time="2020-02-20T17:39:48Z" level=debug msg="No pre-check command supplied. Skipping"
time="2020-02-20T17:39:48Z" level=debug msg="No pre-check command supplied. Skipping"
time="2020-02-20T17:39:48Z" level=debug msg="No pre-check command supplied. Skipping"
time="2020-02-20T17:39:48Z" level=debug msg="Retrieving running containers"
time="2020-02-20T17:39:48Z" level=debug msg="Pulling 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest for /eager_swartz"
time="2020-02-20T17:39:48Z" level=debug msg="No credentials for 618804428340.dkr.ecr.us-east-1.amazonaws.com in /config.json"
time="2020-02-20T17:39:48Z" level=debug msg="Got auth value: "
time="2020-02-20T17:39:48Z" level=debug msg="Got image name: 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest"
time="2020-02-20T17:39:48Z" level=debug msg="No authentication credentials found for 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest"
time="2020-02-20T17:39:48Z" level=debug msg="No new images found for /eager_swartz"
time="2020-02-20T17:39:48Z" level=debug msg="Pulling k8s.gcr.io/pause:latest for /pause"
time="2020-02-20T17:39:48Z" level=debug msg="No credentials for k8s.gcr.io in /config.json"
time="2020-02-20T17:39:48Z" level=debug msg="Got auth value: "
time="2020-02-20T17:39:48Z" level=debug msg="Got image name: k8s.gcr.io/pause:latest"
time="2020-02-20T17:39:48Z" level=debug msg="No authentication credentials found for k8s.gcr.io/pause:latest"
time="2020-02-20T17:39:49Z" level=debug msg="No new images found for /pause"
time="2020-02-20T17:39:49Z" level=debug msg="Pulling containrrr/watchtower:latest for /watchtower"
time="2020-02-20T17:39:49Z" level=debug msg="No credentials for containrrr in /config.json"
time="2020-02-20T17:39:49Z" level=debug msg="Got auth value: "
time="2020-02-20T17:39:49Z" level=debug msg="Got image name: containrrr/watchtower:latest"
time="2020-02-20T17:39:49Z" level=debug msg="No authentication credentials found for containrrr/watchtower:latest"
time="2020-02-20T17:39:49Z" level=debug msg="No new images found for /watchtower"
time="2020-02-20T17:39:49Z" level=debug msg="This is the watchtower container /watchtower"
time="2020-02-20T17:39:49Z" level=debug msg="Retrieving running containers"
time="2020-02-20T17:39:49Z" level=debug msg="No post-check command supplied. Skipping"
time="2020-02-20T17:39:49Z" level=debug msg="No post-check command supplied. Skipping"
time="2020-02-20T17:39:49Z" level=debug msg="No post-check command supplied. Skipping"
time="2020-02-20T17:39:49Z" level=debug msg="Scheduled next run: 2020-02-20 17:40:48 +0000 UTC"

5. Restart Watchtower.

[root@ip-10-0-2-15 vagrant]# docker restart watchtower
watchtower

6. Observe that Watchtower can check the private repository for updates.

time="2020-02-20T17:43:16Z" level=debug msg="Checking containers for updated images"                                       
time="2020-02-20T17:43:16Z" level=debug msg="Retrieving running containers"                                                
time="2020-02-20T17:43:16Z" level=debug msg="No pre-check command supplied. Skipping"                                      
time="2020-02-20T17:43:16Z" level=debug msg="No pre-check command supplied. Skipping"                                      
time="2020-02-20T17:43:16Z" level=debug msg="No pre-check command supplied. Skipping"                                      
time="2020-02-20T17:43:16Z" level=debug msg="Retrieving running containers"                                                
time="2020-02-20T17:43:16Z" level=debug msg="Pulling 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest for /eager_swartz"
time="2020-02-20T17:43:16Z" level=debug msg="Loaded auth credentials <redacted> https://618804428340.dkr.ecr.us-east-1.amazonaws.com  } from /config.json"
time="2020-02-20T17:43:16Z" level=debug msg="Got auth value: <redacted>"                                                
time="2020-02-20T17:43:16Z" level=debug msg="Got image name: 618804428340.dkr.ecr.us-east-1.amazonaws.com/gocd_agent_build_container:latest"
time="2020-02-20T17:43:16Z" level=debug msg="No new images found for /eager_swartz"                                     
time="2020-02-20T17:43:16Z" level=debug msg="Pulling k8s.gcr.io/pause:latest for /pause"                                   
time="2020-02-20T17:43:16Z" level=debug msg="No credentials for k8s.gcr.io in /config.json"                                
time="2020-02-20T17:43:16Z" level=debug msg="Got auth value: "                                                             
time="2020-02-20T17:43:16Z" level=debug msg="Got image name: k8s.gcr.io/pause:latest"                                      
time="2020-02-20T17:43:16Z" level=debug msg="No authentication credentials found for k8s.gcr.io/pause:latest"              
time="2020-02-20T17:43:17Z" level=debug msg="No new images found for /pause"                                               
time="2020-02-20T17:43:17Z" level=debug msg="Pulling containrrr/watchtower:latest for /watchtower"                         
time="2020-02-20T17:43:17Z" level=debug msg="No credentials for containrrr in /config.json"                                
time="2020-02-20T17:43:17Z" level=debug msg="Got auth value: "                                                             
time="2020-02-20T17:43:17Z" level=debug msg="Got image name: containrrr/watchtower:latest"                                 
time="2020-02-20T17:43:17Z" level=debug msg="No authentication credentials found for containrrr/watchtower:latest"         
time="2020-02-20T17:43:18Z" level=debug msg="No new images found for /watchtower"                                          
time="2020-02-20T17:43:18Z" level=debug msg="This is the watchtower container /watchtower"                                 
time="2020-02-20T17:43:18Z" level=debug msg="Retrieving running containers"                                                
time="2020-02-20T17:43:18Z" level=debug msg="No post-check command supplied. Skipping"                                     
time="2020-02-20T17:43:18Z" level=debug msg="No post-check command supplied. Skipping"                                     
time="2020-02-20T17:43:18Z" level=debug msg="No post-check command supplied. Skipping"                                     
time="2020-02-20T17:43:18Z" level=debug msg="Scheduled next run: 2020-02-20 17:44:16 +0000 UTC"

Expected behavior
Watchtower should reread config.json if it changes. This will prevent having to restart Watchtower if credentials change, or new credentials are added/removed with login/logout.

Screenshots
N/A

Environment

• Platform: CentOS 7
• Architecture: x86_64
• Docker version: 1.13.1

Logs from running watchtower with the --debug option
Shown above.

Additional context

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[ghost]

I am also having this issue. I have been debugging it for the last few days and like you have found when the file is updated on the host it is not updated on the watchtower server. I read that as the file is being changed by something like docker the bind mount is being broken.

A potential workaround for this being broken is to mount a directory rather than a single file however in this case this is not possible due to the config.json having to be at the root of the watchtower image. If config.json was stored inside a folder then this workaround may work.

[simskij]

Thank you for the report!

Feel free to have a go at implementing this change! I'd be happy to assist with reviews and direction to the best of my ability. 🙏🏼

[tammert]

So I did a little digging and I found the following:

• watchtower will read the config.json file each scan it does, so it picks up changes if propagated to the container
• regarding propagation: File mount does not update with changes from host moby/moby#15793 (comment)

I tested this out: when I edit the file with vim on the host, I still see the 'old' config.json in my container. If I edit with nano on the host, however, the file is correctly updated in the container too! If you want to achieve the same with vim, you can use set nobackup nowritebackup (either in the running vim or in your .vimrc).

However, as far as docker login goes, I'm not sure if you can get it to emulate that behavior. The only way to really solve this would be to mount the directory containing config.json into watchtower (as proposed by @icabbi-joegarlick), but that would require a new location for config.json in the container and I think that's a pretty backwards-incompatible change which will affect all users mounting config.json. In that regard I think the inconvenience of this change is higher than maybe an extra restart in this scenario, but perhaps @simskij can decide on that.

Actually, an alternative suggestion would be to make the location configurable by the user, but still defaulting to /config.json. Perhaps that would solve this issue but does not inconvenience all current users?

[Happyfeet01]

hi, i have the same issue. it loads only the old login data.

no idea what can i do. docker login at the host works fine.

[simskij]

@tammert Did you try out the approach with symlinking the config file and mounting the symlink rather than the immediate file, as described in the issue you linked? That seems to have solved it for some. 🤷‍♂️ As in:

$ ln -s ~/.docker/config.json ~/.docker/linked-config.json

and then mounting ~/.docker/linked-config.json instead. The reasoning behind why this would work, is that the inode (which docker uses to track changes to mounted files) is preserved for the symlink, even if the actual inode of the config.json file is broken.

[tammert]

@simskij haven't checked that actually, but I'm sure it would work as you described. So I think if we want to solve this issue, we can either:
a) describe the symlink workaround in the docs
b) describe the restart workaround in the docs
c) make the location configurable by the user, which would allow mounting of a directory containing the config.json

I don't have a strong preference either way, but I wouldn't mind implementing either of the 3. Any thoughts?

[simskij]

I'm leaning towards option a, as that adds no additional complexity to watchtower, relies on well-established OS features and once set-up, gets rid of the manual labor. Would be awesome if you wanted to take lead on that 👍

[tammert]

@simskij PR is up for option a!

[ghost]

To avoid important communication to get lost in a closed issues no one monitors, I'll go ahead and lock this issue. If you want to continue the discussion, please open a new issue. Thank you! 🙏🏼