upload doesn't work while using an SSL Client Certificate

[sebpapillon]

The bug

upload doesn't work when I'm using an SSL Client Certificate, works well to connect to the server, download, display pictures but will not upload anything. I'm using haproxy and the error message is quite clear :

fd[0x21] OpenSSL error[0xa0000c7] tls_process_client_certificate: peer did not return a certificate

The OS that Immich Server is running on

Debian 12

Version of Immich Server

v1.112.1

Version of Immich Mobile App

v1.112.1

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1.open the mobile app
2.long press on an image marked as not uploaded 
3.select upload
...

Relevant log output

from haproxy in debug mode
fd[0x21] OpenSSL error[0xa0000c7] tls_process_client_certificate: peer did not return a certificate

Additional information

No response

[tgutzler]

I was having upload issues after putting nginx in front of my immich server.
The root cause was different - I had forgotten to increase the default body size, which caused nginx to throw an error.

[PottiMc]

As my issue was unfortunately moved to discussions I can tell you I have the exact same issue but don't use a self-hosted reverse proxy. Instead I use a Cloudflare Tunnel. I can't find any logs there but my issue is exactly the same. Can do everything in instance, only mobile uploading is not working.
When I switched the app to the local IP, everything is working fine.

[RogerHartmann]

I have a similar issue with nginx and iOS. Features beside of uploading seem to work fine.
What I am wondering is that the iOS client is also giving a hint about iCloud when I am connected via reverse proxy and trying to do an upload (and the upload fails). When I am directly connected to immich, I do not see that hint and the upload is working.
No idea if that is somehow related...

Update: when trying to gather some information for debugging, I got the update working (Update2: but not reliably).
After first login, it failed again. Find the iCloud download screenshot

Then, when I wanted to re-trigger the upload and record some nginx logs, I killed the iOS app. After restarting the iOS app, the immich upload was working.

I wonder how iCloud download is related to image upload. The photo was just taken locally and never uploaded to iCloud.

Update2: it is not really stable. Now, after some more minutes, upload is getting stuck again

[sebpapillon]

the problem seems to have solved itself for whatever reason unknown to me.

[neothematrix]

thanks @bo0tzz for linking my issue #12067 here, however, in my case, the "foreground" backup is working properly, only the background backup is not, and the reason seems to be that background backup process doesn't present the SSL Client certificate to the server.

[neothematrix]

to complement my comment: if I disable the mTLS requirement on cloudflare (that of course, I'd like to keep), background backup works flawlessly.

[PottiMc]

the problem seems to have solved itself for whatever reason unknown to me.

Not for me unfortunetaly.

[mmomjian]

thanks @bo0tzz for linking my issue #12067 here, however, in my case, the "foreground" backup is working properly, only the background backup is not, and the reason seems to be that background backup process doesn't present the SSL Client certificate to the server.

These problems are similar enough to be tracked in one issue.

[neothematrix]

the problem seems to have solved itself for whatever reason unknown to me.

Not for me unfortunetaly.

@PottiMc are you also using mTLS on cloudflare?

[PottiMc]

No, I am not. Only a cloudflare tunnel to my instance. Everything worked fine before updating to 1.112. I still don't know why my issue was moved to discussions, but you can find all my comments here: #11873

[neothematrix]

maybe @yjiang-c as owner of the mTLS PR implementation can have a look here?

[yjiang-c]

I cannot reproduce the issue with my iPhone and Ngnix reverse proxy server. I checked the source code and cannot find any different between upload (POST) and download (GET), both use the same http client stack.

[PottiMc]

As you can see in my discussion #11873 my issue got solved because cloudflare changed something in their Managed Firewall Ruleset. @sebpapillon maybe you are running a firewall aswell?

[neothematrix]

I cannot reproduce the issue with my iPhone and Ngnix reverse proxy server. I checked the source code and cannot find any different between upload (POST) and download (GET), both use the same http client stack.

thanks for checking @yjiang-c ! I'm configuring an iPhone to do the same test, just for clarity, did you check that the "automatic background backup" is working properly with mTLS certificate? because in my scenario, the foreground backup and the manual upload is working without any issue, but when the background upload triggers, I see the requests sent do not contain the mTLS certificate.

[yjiang-c]

@neothematrix Yeah, I can confirm that background backup with SSL client certificate does not work. In addition, I think that background backup does not work with server with self signed certificate.

Background back runs in different process/zone in which http SSL options are not set properly. I will create MR to fix it.

[neothematrix]

Thanks a lot @yjiang-c that's really appreciated!

While you're at it, any chance you could look into a similar issue? #5553 (comment)

However, not sure it's very easy to fix and it's probably platform dependent, it looks like the same SSL options are not passed to external players used to playback videos.

Again, thank you so much and I'll be very happy to test anything that need to be tested!

[yjiang-c]

@neothematrix #5553 is not easy to be resolved. The video player used in immich mobile is a flutter plugin, which does not support customised HTTPS options. In immich side, we can only fix it by totally replacing the video player and rewrite a new video player from scratch.

[neothematrix]

@yjiang-c thanks a lot for the PR, hopefully it will be accepted and merged soon!
thanks a lot for also checking #5553 I had a feeling it was not that easy, perhaps we need @alextran1502 attention there, however, the easy workaround is using webbrowser instead of Immich App to playback the videos.

[neothematrix]

tested with the updated app on Android and now it's working flawlessly with Client Certificate, thanks a lot @yjiang-c and @alextran1502 !
now to fix #5553 :)