From c94e1c1c10acaff13813e8e073fc7ebf9e3bfac2 Mon Sep 17 00:00:00 2001 From: mmomjian <50788000+mmomjian@users.noreply.github.com> Date: Sun, 19 May 2024 01:50:50 -0400 Subject: [PATCH 01/10] docs for HC --- docs/docs/FAQ.mdx | 105 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 5458a226200bf..636edd13ba0e7 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -360,6 +360,111 @@ cap_drop: +### How can I monitor the health of the Immich containers? + +All of the containers have health monitoring scripts that are not enabled by default. To enable them, add the following to your `docker-compose.yml`. + +```diff +services: + immich-server: + ... ++ healthcheck: ++ test: npm run healthcheck + + immich-machine-learning: + ... ++ healthcheck: ++ test: python3 healthcheck.py + + redis: + ... ++ healthcheck: ++ test: redis-cli ping + + database: + ... ++ healthcheck: ++ test: ["CMD-SHELL", "pg_isready", "--dbname=${DB_DATABASE_NAME}"] +``` + +### How can I monitor the health of the Immich containers? + +All of the containers have health monitoring scripts that are not enabled by default. To enable them, add the following to your `docker-compose.yml`. + +```diff +๐Ÿ‹ What is a docker healthcheck +the docker spec has a way to determine if a service is healthy. This is useful for servers. The command docker ps can be used to get a quick summary of if a container is responding positively. + +Platforms like AWS have built in integration for this spec and show the health status in their UI when running containers in ECS Fargate. + +There probably are other benefits but I only know of those. I personally like to use the lazydocker cli to manage my docker services. Which also displays if they are passing healthchecks + +lazy + +๐Ÿ—๏ธ My solution +One thing to know about the docker healthcheck is it uses CLI on the image to perform it. This can be tricky to accomplish in an era of distroless images. The images that Immich puts out seem to be in that category (which is a good thing). The following common CLI that allow docker healthchecks to easily be done are absent on the image for both immich_server and immich_machine_learning + +common commands used for Docker healthchecks + +curl +wget +netcat or nc +pgrep +This means to support docker healthchecks you have to go one of two routes. Add wget from the busybox image. Or use a programming language to script a healthcheck command. + +If the end image has node or python, like is the case here, then the later is the most slim and secure way to do this. + +So, that's what I've added for the two images + +๐Ÿงช Tests +Lint +I ran npm run lint but got the following error + +ignore_lint + +Healthchecks do need to follow the Docker spec and return exactly what Docker wants. + +0: success - the container is healthy and ready for use +1: unhealthy - the container isn't working correctly +2: reserved - don't use this exit code +So, since using process.exit() is a requirement. I added this to the ignore patterns of eslint instead of resolving it. + +Tests +passing +pass + +Something I haven't tested +Healthchecks by default will run every 30 seconds. This in some apps could cause an excess of logs. I did not check for this. If anyone wants to give that concern a look. That would be appreciated. + +๐Ÿ˜ถโ€๐ŸŒซ๏ธKeep Optional +It would be possible to add these to the installation docker-compose.yml file. But I don't think it would be good to force this out (by adding it to /docker/docker-compose.yml). It can always be something that people optionally add onto their compose file if they want it (it's more for people who know what they are doing anyways). For example postgres and redis have a built in way to do healthchecks. Which I'm adding to my docker compose file. + +I guess something in the docs would be useful for this. But curious what the maintainers think on this. + +An example of what my compose file looks like with healthchecks + +services: + immich-server: + ... ++ healthcheck: ++ test: npm run healthcheck + + immich-machine-learning: + ... ++ healthcheck: ++ test: python3 healthcheck.py + + redis: + ... ++ healthcheck: ++ test: redis-cli ping + + database: + ... ++ healthcheck: ++ test: ["CMD-SHELL", "pg_isready", "--dbname=${DB_DATABASE_NAME}"] +``` + ### How can I purge data from Immich? Data for Immich comes in two forms: From 405a2fc6affe45ae418e1bc34af49b3cb73fa5ea Mon Sep 17 00:00:00 2001 From: Matthew Momjian <50788000+mmomjian@users.noreply.github.com> Date: Sun, 19 May 2024 01:51:45 -0400 Subject: [PATCH 02/10] Update FAQ.mdx --- docs/docs/FAQ.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 636edd13ba0e7..de52f05813f2f 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -362,7 +362,7 @@ cap_drop: ### How can I monitor the health of the Immich containers? -All of the containers have health monitoring scripts that are not enabled by default. To enable them, add the following to your `docker-compose.yml`. +All of the containers have health monitoring scripts that are disabled by default. To enable them, add the following to your `docker-compose.yml`. ```diff services: From b8cd74e4fc944ca3ff991b61f8b60919b5e61b5f Mon Sep 17 00:00:00 2001 From: Matthew Momjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 10:34:56 -0400 Subject: [PATCH 03/10] Update FAQ.mdx --- docs/docs/FAQ.mdx | 78 ----------------------------------------------- 1 file changed, 78 deletions(-) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index de52f05813f2f..56785c388ade2 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -387,84 +387,6 @@ services: + test: ["CMD-SHELL", "pg_isready", "--dbname=${DB_DATABASE_NAME}"] ``` -### How can I monitor the health of the Immich containers? - -All of the containers have health monitoring scripts that are not enabled by default. To enable them, add the following to your `docker-compose.yml`. - -```diff -๐Ÿ‹ What is a docker healthcheck -the docker spec has a way to determine if a service is healthy. This is useful for servers. The command docker ps can be used to get a quick summary of if a container is responding positively. - -Platforms like AWS have built in integration for this spec and show the health status in their UI when running containers in ECS Fargate. - -There probably are other benefits but I only know of those. I personally like to use the lazydocker cli to manage my docker services. Which also displays if they are passing healthchecks - -lazy - -๐Ÿ—๏ธ My solution -One thing to know about the docker healthcheck is it uses CLI on the image to perform it. This can be tricky to accomplish in an era of distroless images. The images that Immich puts out seem to be in that category (which is a good thing). The following common CLI that allow docker healthchecks to easily be done are absent on the image for both immich_server and immich_machine_learning - -common commands used for Docker healthchecks - -curl -wget -netcat or nc -pgrep -This means to support docker healthchecks you have to go one of two routes. Add wget from the busybox image. Or use a programming language to script a healthcheck command. - -If the end image has node or python, like is the case here, then the later is the most slim and secure way to do this. - -So, that's what I've added for the two images - -๐Ÿงช Tests -Lint -I ran npm run lint but got the following error - -ignore_lint - -Healthchecks do need to follow the Docker spec and return exactly what Docker wants. - -0: success - the container is healthy and ready for use -1: unhealthy - the container isn't working correctly -2: reserved - don't use this exit code -So, since using process.exit() is a requirement. I added this to the ignore patterns of eslint instead of resolving it. - -Tests -passing -pass - -Something I haven't tested -Healthchecks by default will run every 30 seconds. This in some apps could cause an excess of logs. I did not check for this. If anyone wants to give that concern a look. That would be appreciated. - -๐Ÿ˜ถโ€๐ŸŒซ๏ธKeep Optional -It would be possible to add these to the installation docker-compose.yml file. But I don't think it would be good to force this out (by adding it to /docker/docker-compose.yml). It can always be something that people optionally add onto their compose file if they want it (it's more for people who know what they are doing anyways). For example postgres and redis have a built in way to do healthchecks. Which I'm adding to my docker compose file. - -I guess something in the docs would be useful for this. But curious what the maintainers think on this. - -An example of what my compose file looks like with healthchecks - -services: - immich-server: - ... -+ healthcheck: -+ test: npm run healthcheck - - immich-machine-learning: - ... -+ healthcheck: -+ test: python3 healthcheck.py - - redis: - ... -+ healthcheck: -+ test: redis-cli ping - - database: - ... -+ healthcheck: -+ test: ["CMD-SHELL", "pg_isready", "--dbname=${DB_DATABASE_NAME}"] -``` - ### How can I purge data from Immich? Data for Immich comes in two forms: From ea07199a13e1847fb29381ef3756bd110458d1fe Mon Sep 17 00:00:00 2001 From: Matthew Momjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 12:21:44 -0400 Subject: [PATCH 04/10] Update FAQ.mdx --- docs/docs/FAQ.mdx | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 56785c388ade2..9760ef3275854 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -362,29 +362,22 @@ cap_drop: ### How can I monitor the health of the Immich containers? -All of the containers have health monitoring scripts that are disabled by default. To enable them, add the following to your `docker-compose.yml`. +All of the containers have health monitoring scripts. These are enabled by default in `immich-server` and `immich-machine-learning`. To enable them for the other containers, add the following to your `docker-compose.yml`. ```diff services: - immich-server: - ... -+ healthcheck: -+ test: npm run healthcheck - - immich-machine-learning: - ... -+ healthcheck: -+ test: python3 healthcheck.py - redis: ... + healthcheck: -+ test: redis-cli ping ++ test: redis-cli ping || exit 1 database: ... + healthcheck: -+ test: ["CMD-SHELL", "pg_isready", "--dbname=${DB_DATABASE_NAME}"] ++ test: test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command="SELECT checksum_failures FROM pg_stat_database WHERE datname = '${DB_DATABASE_NAME}'")"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 ++ interval: 5m ++ start_interval: 30s ++ start_period: 5m ``` ### How can I purge data from Immich? From 8304cc0536ac946e55d7fda5276b9642709adef5 Mon Sep 17 00:00:00 2001 From: Matthew Momjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 12:36:10 -0400 Subject: [PATCH 05/10] Update FAQ.mdx --- docs/docs/FAQ.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 9760ef3275854..f5a09900a05d7 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -374,7 +374,7 @@ services: database: ... + healthcheck: -+ test: test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command="SELECT checksum_failures FROM pg_stat_database WHERE datname = '${DB_DATABASE_NAME}'")"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 ++ test: test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 + interval: 5m + start_interval: 30s + start_period: 5m From 0d141ee7894186f94381e6b6c04141e58526a0b0 Mon Sep 17 00:00:00 2001 From: Matthew Momjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 12:59:05 -0400 Subject: [PATCH 06/10] Update FAQ.mdx --- docs/docs/FAQ.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index f5a09900a05d7..3ba73298ebe40 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -374,7 +374,7 @@ services: database: ... + healthcheck: -+ test: test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 ++ test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 + interval: 5m + start_interval: 30s + start_period: 5m From 44ac106672afc25b99998516566f5d8d4e41e50d Mon Sep 17 00:00:00 2001 From: mmomjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 13:07:01 -0400 Subject: [PATCH 07/10] HCs -> docker compose --- docker/docker-compose.dev.yml | 7 +++++++ docker/docker-compose.prod.yml | 10 +++++++++- docker/docker-compose.yml | 9 ++++++++- docs/docs/FAQ.mdx | 20 -------------------- 4 files changed, 24 insertions(+), 22 deletions(-) diff --git a/docker/docker-compose.dev.yml b/docker/docker-compose.dev.yml index f18a563b34561..21128f8aca822 100644 --- a/docker/docker-compose.dev.yml +++ b/docker/docker-compose.dev.yml @@ -82,6 +82,8 @@ services: redis: container_name: immich_redis image: redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + healthcheck: + test: redis-cli ping || exit 1 database: container_name: immich_postgres @@ -97,6 +99,11 @@ services: - ${UPLOAD_LOCATION}/postgres:/var/lib/postgresql/data ports: - 5432:5432 + healthcheck: + test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 + interval: 5m + start_interval: 30s + start_period: 5m command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] # set IMMICH_METRICS=true in .env to enable metrics diff --git a/docker/docker-compose.prod.yml b/docker/docker-compose.prod.yml index fc62ef0c5b29c..ae5f8186afdf8 100644 --- a/docker/docker-compose.prod.yml +++ b/docker/docker-compose.prod.yml @@ -12,12 +12,12 @@ services: - /etc/localtime:/etc/localtime:ro env_file: - .env - restart: always ports: - 2283:3001 depends_on: - redis - database + restart: always immich-machine-learning: container_name: immich_machine_learning @@ -39,6 +39,8 @@ services: redis: container_name: immich_redis image: redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + healthcheck: + test: redis-cli ping || exit 1 restart: always database: @@ -55,7 +57,13 @@ services: - ${UPLOAD_LOCATION}/postgres:/var/lib/postgresql/data ports: - 5432:5432 + healthcheck: + test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 + interval: 5m + start_interval: 30s + start_period: 5m command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] + restart: always # set IMMICH_METRICS=true in .env to enable metrics immich-prometheus: diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 16d628258eb0d..62173a2ba8e17 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -41,6 +41,8 @@ services: redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + healthcheck: + test: redis-cli ping || exit 1 restart: always database: @@ -53,8 +55,13 @@ services: POSTGRES_INITDB_ARGS: '--data-checksums' volumes: - ${DB_DATA_LOCATION}:/var/lib/postgresql/data - restart: always + healthcheck: + test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 + interval: 5m + start_interval: 30s + start_period: 5m command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] + restart: always volumes: model-cache: diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 3ba73298ebe40..5458a226200bf 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -360,26 +360,6 @@ cap_drop: -### How can I monitor the health of the Immich containers? - -All of the containers have health monitoring scripts. These are enabled by default in `immich-server` and `immich-machine-learning`. To enable them for the other containers, add the following to your `docker-compose.yml`. - -```diff -services: - redis: - ... -+ healthcheck: -+ test: redis-cli ping || exit 1 - - database: - ... -+ healthcheck: -+ test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT SUM(checksum_failures) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 -+ interval: 5m -+ start_interval: 30s -+ start_period: 5m -``` - ### How can I purge data from Immich? Data for Immich comes in two forms: From 759b8cadc2eee8172e70731899ba20aecef65262 Mon Sep 17 00:00:00 2001 From: mmomjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 13:09:41 -0400 Subject: [PATCH 08/10] bump redis to match main branch --- docker/docker-compose.dev.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/docker-compose.dev.yml b/docker/docker-compose.dev.yml index 21128f8aca822..a2b8bf0bb4e51 100644 --- a/docker/docker-compose.dev.yml +++ b/docker/docker-compose.dev.yml @@ -81,7 +81,7 @@ services: redis: container_name: immich_redis - image: redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + image: redis:6.2-alpine@sha256:c0634a08e74a4bb576d02d1ee993dc05dba10e8b7b9492dfa28a7af100d46c01 healthcheck: test: redis-cli ping || exit 1 From 286f582006b7a998b62ed95fb88b399ce53155a6 Mon Sep 17 00:00:00 2001 From: mmomjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 13:10:44 -0400 Subject: [PATCH 09/10] conflicts --- docker/docker-compose.prod.yml | 2 +- docker/docker-compose.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/docker-compose.prod.yml b/docker/docker-compose.prod.yml index ae5f8186afdf8..89bfb80ab603b 100644 --- a/docker/docker-compose.prod.yml +++ b/docker/docker-compose.prod.yml @@ -38,7 +38,7 @@ services: redis: container_name: immich_redis - image: redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + image: docker.io/redis:6.2-alpine@sha256:c0634a08e74a4bb576d02d1ee993dc05dba10e8b7b9492dfa28a7af100d46c01 healthcheck: test: redis-cli ping || exit 1 restart: always diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 62173a2ba8e17..6361aa4c1d0c1 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -40,7 +40,7 @@ services: redis: container_name: immich_redis - image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 + image: docker.io/redis:6.2-alpine@sha256:c0634a08e74a4bb576d02d1ee993dc05dba10e8b7b9492dfa28a7af100d46c01 healthcheck: test: redis-cli ping || exit 1 restart: always From 0cf0b7190d1e4903101f075b42963066e850f678 Mon Sep 17 00:00:00 2001 From: mmomjian <50788000+mmomjian@users.noreply.github.com> Date: Tue, 21 May 2024 13:11:45 -0400 Subject: [PATCH 10/10] more conflicts lol --- docker/docker-compose.prod.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/docker-compose.prod.yml b/docker/docker-compose.prod.yml index 89bfb80ab603b..fb17d4293283c 100644 --- a/docker/docker-compose.prod.yml +++ b/docker/docker-compose.prod.yml @@ -38,7 +38,7 @@ services: redis: container_name: immich_redis - image: docker.io/redis:6.2-alpine@sha256:c0634a08e74a4bb576d02d1ee993dc05dba10e8b7b9492dfa28a7af100d46c01 + image: redis:6.2-alpine@sha256:c0634a08e74a4bb576d02d1ee993dc05dba10e8b7b9492dfa28a7af100d46c01 healthcheck: test: redis-cli ping || exit 1 restart: always