watchtower hooks not working #1004

dev2021-ctrl · 2021-06-28T15:42:05Z

Hi

I am using pre update hooks in watchtower but the below commands does not even trigger the shell script.
Basically i want to skip update based on shell script
Also this env variable set WATCHTOWER_LIFECYCLE_HOOKS to true

docker run  -d --label=com.centurylinklabs.watchtower.lifecycle.pre-check="/dump-data.sh" \
--restart always \
--name watchtower \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower \
--cleanup \
--interval 10 --enable-lifecycle-hooks \
test

Pleae help a bit urgent
I am pretty sure the latest watchtower build here https://github.com/containrrr/watchtower supports this skipping update feature

Logs :

time="2021-06-28T15:21:17Z" level=info msg="Watchtower 1.3.0\nUsing no notifications\nOnly checking containers with name \"validator\"\nScheduling first run: 2021-06-28 15:21:27 +0000 UTC\nNote that the first check will be performed in 9 seconds"

The text was updated successfully, but these errors were encountered:

github-actions · 2021-06-28T15:42:40Z

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

dev2021-ctrl · 2021-06-28T15:44:06Z

Already read it and posted the bug above
please help

piksel · 2021-06-29T09:32:58Z

Look at the examples in the docs for examples of how to use the hooks:
https://containrrr.dev/watchtower/lifecycle-hooks/

You are currently telling watchtower to only check the container "test" and run any lifecycle-hooks on it before/after checking/updating it.
But the label with the hook is passed to the watchtower container (which neither contain sh, nor the specified script dump-data.sh).

piksel · 2021-06-29T09:41:28Z

I am pretty sure the latest watchtower build here https://github.com/containrrr/watchtower supports this skipping update feature

I think you are referring to #793, no? That has been merged, but it's not released yet.
You can use containrrr/watchtower:latest-dev to try it out!

dev2021-ctrl · 2021-06-29T09:47:00Z

ok thanks I am now creating shell script inside the container and running it now Will update in few mins

…

On Tue, Jun 29, 2021 at 3:11 PM nils måsén ***@***.***> wrote: I am pretty sure the latest watchtower build here https://github.com/containrrr/watchtower supports this skipping update feature I think you are referring to #793 <#793>, no? That has been merged, but it's not released yet. You can use containrrr/watchtower:latest-dev to try it out! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1004 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT3CT6BDGI55NAQ2SCGGVJTTVGINDANCNFSM47OHL4NA> .

dev2021-ctrl · 2021-06-29T09:48:30Z

I assume with latest-dev if the label pre update check fails due to exit code other than 0 , the watchtower wont update and will retry after set interval. Correct

…

On Tue, Jun 29, 2021 at 3:11 PM nils måsén ***@***.***> wrote: I am pretty sure the latest watchtower build here https://github.com/containrrr/watchtower supports this skipping update feature I think you are referring to #793 <#793>, no? That has been merged, but it's not released yet. You can use containrrr/watchtower:latest-dev to try it out! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1004 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT3CT6BDGI55NAQ2SCGGVJTTVGINDANCNFSM47OHL4NA> .

dev2021-ctrl · 2021-06-29T09:56:18Z

@piksel can you please let me know how to create that pre update scrupt inside container.
I tried
docker run -it "containrrr/watchtower:latest-dev" bash
but not workin

dev2021-ctrl · 2021-06-29T10:20:30Z

Look at the examples in the docs for examples of how to use the hooks:
https://containrrr.dev/watchtower/lifecycle-hooks/

You are currently telling watchtower to only check the container "test" and run any lifecycle-hooks on it before/after checking/updating it.
But the label with the hook is passed to the watchtower container (which neither contain sh, nor the specified script dump-data.sh).

Can you please share the complete commands.I am confused

piksel · 2021-06-29T10:40:31Z

I am not sure what you are trying to do, but to run a container that will have it's script executed at the start of every update-session, you can use the following:

docker run -d \
  --label=com.centurylinklabs.watchtower.lifecycle.pre-check="/dump-data.sh" \
  --name validator \
  your-validator/image:latest

where your-validator/image contains the /dump-data.sh script you want to run

Then run watchtower(-dev) with lifecycle hooks enabled:

docker run -d  \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower:latest-dev \
  --debug \
  --interval 10 --enable-lifecycle-hooks \
  validator

dev2021-ctrl · 2021-06-29T10:46:20Z

i wil check thanks

…

On Tue, Jun 29, 2021 at 4:10 PM nils måsén ***@***.***> wrote: I am not sure what you are trying to do, but to run a container that will have it's script executed at the start of every update-session, you can use the following: docker run -d \ --label=com.centurylinklabs.watchtower.lifecycle.pre-check="/dump-data.sh" \ --name validator \ your-validator/image:latest where your-validator/image contains the /dump-data.sh script you want to run Then run watchtower(-dev) with lifecycle hooks enabled: docker run -d \ --name watchtower \ -v /var/run/docker.sock:/var/run/docker.sock \ containrrr/watchtower-dev \ --debug \ --interval 10 --enable-lifecycle-hooks \ validator — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1004 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT3CT6GEB7U63PLCPKHXO5TTVGPKRANCNFSM47OHL4NA> .

dev2021-ctrl · 2021-06-29T10:52:12Z

I am not sure what you are trying to do, but to run a container that will have it's script executed at the start of every update-session, you can use the following:
docker run -d \
  --label=com.centurylinklabs.watchtower.lifecycle.pre-check="/dump-data.sh" \
  --name validator \
  your-validator/image:latest
where your-validator/image contains the /dump-data.sh script you want to run

Then run watchtower(-dev) with lifecycle hooks enabled:
docker run -d  \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower-dev \
  --debug \
  --interval 10 --enable-lifecycle-hooks \
  validator

Basicay i want to update my validator image based on the below condition
The condition will be:
if (docker exec validator miner info in_consensus) = false
then (watchtower can update)
else (watchtower can NOT update)

I am wondering I can create dump-data.sh script and add to validator container.
If it is false then I can make shell script return non zero code and the update wil be skipped by watchtower.

I assume the above logic will work

dev2021-ctrl · 2021-06-29T11:20:23Z

I am using above commands but cannot see any labels present when doing inspect
docker image inspect --format='' "quay.io/team-helium/validator:latest-validator-amd64"

This is the docker for validator
docker run -d --label=com.centurylinklabs.watchtower.lifecycle.pre-update="/dump.sh" --init
--restart always
--publish 2154:2154/tcp
--name validator
--mount type=bind,source=$HOME/validator_data,target=/var/data
quay.io/team-helium/validator:latest-validator-amd64

please advise
a bit urgent

piksel · 2021-06-29T11:39:00Z

you are putting the labels on the container, not the image:

❯ docker container inspect validator | jq .[0].Config.Labels
{
  "com.centurylinklabs.watchtower.lifecycle.pre-check": "/dump.sh",
  "desktop.docker.io/wsl-distro": "Ubuntu-20.04"
}

dev2021-ctrl · 2021-06-29T11:42:36Z

docker container inspect validator | jq .[0].Config.Labels
Sorry my bad.I can see the labels now but I cannot see my shell script being triggered.
This is my dump.sh shell script
now=$(date +"%T") echo "Current time : $now" >>/tmp/watcher

piksel · 2021-06-29T11:50:27Z

This is my logs with --debug enabled (using a container that didn't have /bin/sh so it fails):

DEBU[0661] Executing pre-check command.
DEBU[0661] Awaiting timeout or completion                exec-id= exit-code=126 running=false
INFO[0661] Command output:
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
ERRO[0661] Command exited with code 126  OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
DEBU[0661] Executing pre-check command.
DEBU[0661] Awaiting timeout or completion                exec-id= exit-code=127 running=false
ERRO[0661] Command exited with code 127
DEBU[0661] No pre-check command supplied. Skipping

You should have something similar.

dev2021-ctrl · 2021-06-29T12:00:39Z

This is my logs with --debug enabled (using a container that didn't have /bin/sh so it fails):

DEBU[0661] Executing pre-check command.
DEBU[0661] Awaiting timeout or completion                exec-id= exit-code=126 running=false
INFO[0661] Command output:
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
ERRO[0661] Command exited with code 126  OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
DEBU[0661] Executing pre-check command.
DEBU[0661] Awaiting timeout or completion                exec-id= exit-code=127 running=false
ERRO[0661] Command exited with code 127
DEBU[0661] No pre-check command supplied. Skipping

You should have something similar.

Can u share the complete command you are running

piksel · 2021-06-29T12:17:18Z

Looking at the source, it's only the pre-update command that cancels the update.
So use

--label=com.centurylinklabs.watchtower.lifecycle.pre-update="/dump-data.sh"

instead.

I'm using this for the validator:

docker run -d --label=com.centurylinklabs.watchtower.lifecycle.pre-update="/usr/bin/false" \
  --name validator nginx

And this for watchtower:

docker run --name watchtower \
  -e CLICOLOR_FORCE=1 \
  -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower:latest-dev \
  --interval 60 --enable-lifecycle-hooks --debug

Result in log:

DEBU[0183] Executing pre-update command.
DEBU[0184] Awaiting timeout or completion                exec-id= exit-code=127 running=false
ERRO[0184] Command exited with code 127
INFO[0184] Skipping container as the pre-update command failed

Note that since I typed /usr/bin/false instead of /bin/false the exit code is 127 instead of 1, but it the outcome is the same

isla6canto · 2021-07-03T15:46:50Z

I am not sure what you are trying to do, but to run a container that will have it's script executed at the start of every update-session, you can use the following:
docker run -d \
  --label=com.centurylinklabs.watchtower.lifecycle.pre-check="/dump-data.sh" \
  --name validator \
  your-validator/image:latest
where your-validator/image contains the /dump-data.sh script you want to run

Then run watchtower(-dev) with lifecycle hooks enabled:
docker run -d  \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower:latest-dev \
  --debug \
  --interval 10 --enable-lifecycle-hooks \
  validator

Hello,

Is it possible to locate the "dump-data.sh" script in "/root".

For example:
--label=com.centurylinklabs.watchtower.lifecycle.pre-check="/root/dump-data.sh" \

piksel · 2021-07-03T21:03:55Z

Sure. I used com.centurylinklabs.watchtower.lifecycle.pre-update="/bin/false" in the example above. As long as the file exists in the container an can be executed by doing sh -c <YOUR_SCRIPT>.

isla6canto · 2021-07-04T23:19:28Z

Got it, could I use "exit 127" to prevent watchtower from updating like this:
--label=com.centurylinklabs.watchtower.lifecycle.pre-update="sh -c 'while [ "$(my_condition)" == "true" ]; do echo "condition true, exiting update"; exit 127; done'" \

piksel · 2021-07-04T23:23:02Z

yeah. any other exit code (except 0) will also cancel the update, but it will be a failure and logged (and notified) as an error.

isla6canto · 2021-07-04T23:27:36Z

Thanks for the help!

calvinbui · 2021-07-05T08:41:15Z

The documentation probably needs an update/clarification

The script/command needs to exist within the target container and not watchtower.
It executes the hook using the Docker client, meaning it defaults to sh -c <YOUR HOOK>
Only an exit code of 0 is considered successful. Any other code and watchtower won't continue to the next hook/step.

piksel · 2021-07-05T11:07:39Z

Yeah. The published documentation is for watchtower:latest though, so there is already some updates in
main/docs/lifecycle-hooks.md that would be published on the next release.

It executes the hook using the Docker client, meaning it defaults to sh -c

Just to clarify, it does use the docker API, but it explicitly adds the sh -c:

watchtower/pkg/container/client.go

Lines 366 to 370 in 7221704

    
           execConfig := types.ExecConfig{ 
        
           	Tty:    true, 
        
           	Detach: false, 
        
           	Cmd:    []string{"sh", "-c", command}, 
        
           }

But it does this to mimic what the docker client does when using docker exec.

calvinbui · 2021-07-05T11:48:00Z

I always through it was baked into the docker API and client, cool. Though isn't the default actually /bin/sh?

Would you take a PR to make it configurable? Not every container I have has /bin/sh, like ones built from the scratch container.

piksel · 2021-07-05T12:06:15Z

Not every container I have has /bin/sh, like ones built from the scratch container.

Like watchtower 😁

But yeah, I think so. Since this could be different per container (some contain no /bin/sh, some are scripts that need to be interpreted). Perhaps it could be done by prefixing the command with ! or something similar? Of course, you could just add sh -c to the script ones' lables, but that is much more difficult to do for all your pre-existing containers...

izmagik · 2021-10-22T11:36:39Z

I can not reproduce it @piksel . In the documentation it says:

The failure of a command to execute, identified by an exit code different than 0, will not prevent watchtower from updating the container. Only an error log statement containing the exit code will be reported.

And I constat this behavior:

time="2021-10-22T10:19:53Z" level=debug msg="Executing pre-update command."
time="2021-10-22T10:19:53Z" level=debug msg="Awaiting timeout or completion" exec-id= exit-code=127 running=false
time="2021-10-22T10:19:53Z" level=error msg="Command exited with code 127."
time="2021-10-22T10:19:53Z" level=error
time="2021-10-22T10:19:53Z" level=info msg="Stopping /application-container (c520c793d067) with SIGTERM"
time="2021-10-22T10:20:00Z" level=debug msg="Skipped another update already running."

So I do not have the same behavior as you have @piksel. Nevertheless, I did see the check in the source code of the watchtower application:

if params.LifecycleHooks {
		if err := lifecycle.ExecutePreUpdateCommand(client, container); err != nil {
			log.Error(err)
			log.Info("Skipping container as the pre-update command failed")
			return err
		}
	}

Ref: https://github.com/containrrr/watchtower/blob/v1.3.0/internal/actions/update.go

My configuration of watchtower is the following:

version: '3.7'

services:
  watchtower:
    container_name: watchtower
    image: containrrr/watchtower:1.3.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config.json:/config.json
    ports:
      - 127.0.0.1:8080:8080
    command: --interval 30 --label-enable --cleanup --enable-lifecycle-hooks --debug
    environment:
      - WATCHTOWER_NOTIFICATIONS=msteams
      - WATCHTOWER_NOTIFICATION_MSTEAMS_HOOK_URL=XXXXXX
      - WATCHTOWER_NOTIFICATION_MSTEAMS_USE_LOG_DATA=true

izmagik · 2021-10-22T12:41:10Z

Update:

I digged in the code. I understand that a new ExTempFail will be introduced and the value of the error code needed will be 75.
@piksel : do you know when it will be integrated in the next release ?

piksel · 2021-10-22T12:43:46Z

Yeah, this has not been released yet and is only available for testing in containrrr/watchtower:latest-dev. We don't have any official date to give you just yet, but it shouldn't be too long until next release.

izmagik · 2021-10-22T13:21:17Z

Ok thanks for your fast response. I will be using latest-dev until then.
The lastest-dev version also fix the issue with the Teams notification.

dev2021-ctrl added Priority: Medium Status: Available Type: Bug Unintended behavior labels Jun 28, 2021

simskij closed this as completed Jan 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

watchtower hooks not working #1004

watchtower hooks not working #1004

dev2021-ctrl commented Jun 28, 2021 •

edited by simskij

Loading

github-actions bot commented Jun 28, 2021

dev2021-ctrl commented Jun 28, 2021

piksel commented Jun 29, 2021

piksel commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 •

edited

Loading

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021 •

edited

Loading

piksel commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 •

edited

Loading

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 •

edited

Loading

isla6canto commented Jul 3, 2021

piksel commented Jul 3, 2021

isla6canto commented Jul 4, 2021

piksel commented Jul 4, 2021

isla6canto commented Jul 4, 2021

calvinbui commented Jul 5, 2021 •

edited

Loading

piksel commented Jul 5, 2021 •

edited

Loading

calvinbui commented Jul 5, 2021

piksel commented Jul 5, 2021

izmagik commented Oct 22, 2021

izmagik commented Oct 22, 2021

piksel commented Oct 22, 2021

izmagik commented Oct 22, 2021

watchtower hooks not working #1004

watchtower hooks not working #1004

Comments

dev2021-ctrl commented Jun 28, 2021 • edited by simskij Loading

github-actions bot commented Jun 28, 2021

dev2021-ctrl commented Jun 28, 2021

piksel commented Jun 29, 2021

piksel commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 • edited Loading

dev2021-ctrl commented Jun 29, 2021 via email

dev2021-ctrl commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021 • edited Loading

piksel commented Jun 29, 2021

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 • edited Loading

dev2021-ctrl commented Jun 29, 2021

piksel commented Jun 29, 2021 • edited Loading

isla6canto commented Jul 3, 2021

piksel commented Jul 3, 2021

isla6canto commented Jul 4, 2021

piksel commented Jul 4, 2021

isla6canto commented Jul 4, 2021

calvinbui commented Jul 5, 2021 • edited Loading

piksel commented Jul 5, 2021 • edited Loading

calvinbui commented Jul 5, 2021

piksel commented Jul 5, 2021

izmagik commented Oct 22, 2021

izmagik commented Oct 22, 2021

Update:

piksel commented Oct 22, 2021

izmagik commented Oct 22, 2021

dev2021-ctrl commented Jun 28, 2021 •

edited by simskij

Loading

piksel commented Jun 29, 2021 •

edited

Loading

dev2021-ctrl commented Jun 29, 2021 •

edited

Loading

piksel commented Jun 29, 2021 •

edited

Loading

piksel commented Jun 29, 2021 •

edited

Loading

calvinbui commented Jul 5, 2021 •

edited

Loading

piksel commented Jul 5, 2021 •

edited

Loading