local file mounted as folder in container on flatcar docker swarm node

[jdall]

Description

[ 1 paragraph concisely describing the bug ]
A deploy of SigNoz to a docker swarm node running flatcar 3815.2.3 fails, because apparently some local configuration files are being mounted as folders instead of files in a container.

Impact

[ 1 sentence detailing the impact this bug is creating for you ]
For now the impact is minor, as we are just in the initial phase of testing SigNoz.

Environment and steps to reproduce

1. Set-up: [ describe the environment Flatcar/Lokomotive/Nebraska etc was running in when encountering the bug; Platform etc. ]
   The Flatcar instance is running on a bare-metal x86_64 server.
   The instance has been initialized as a docker swarm master node.

2. Task: [ describe the task performing when encountering the bug ]
   I was trying to deploy SigNoz to the swarm node, using docker compose and the docker-compose file provided by the SigNoz GitHub repository.

3. Action(s): [ sequence of actions that triggered the bug, see example below ]
   a. git clone -b main https://github.com/SigNoz/signoz.git
   b. cd signoz/deploy/docker-swarm
   c. docker compose -f clickhouse-setup/docker-compose.yaml up -d

4. Error: [describe the error that was triggered]

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "****/docker-swarm/signoz/clickhouse-setup/clickhouse-config.xml" to rootfs at "/etc/clickhouse-server/config.xml": mount ****/docker-swarm/signoz/clickhouse-setup/clickhouse-config.xml:/etc/clickhouse-server/config.xml (via /proc/self/fd/7), flags: 0x5000, data: context="system_u:object_r:container_file_t:s0:c392,c505": not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Expected behavior

[ describe what you expected to happen at 4. above but instead got an error ]
All containers would start running.

Additional information

Please add any information here that does not fit the above format.
The same deployment has been tested towards a docker swarm node running on AlmaLinux 9, where the files are correctly mounted as files in the container, and all containers starts running.

The section in the docker-compose.yaml file, that results in the error, looks as follows:

  clickhouse:
    <<: *clickhouse-defaults
    hostname: clickhouse
    # ports:
    #   - "9000:9000"
    #   - "8123:8123"
    #   - "9181:9181"
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      #- ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
      - ./data/clickhouse/:/var/lib/clickhouse/

In my debugging, I tried modifying the above section slightly, to have a peek into the container filesystem:

   clickhouse:
    <<: *clickhouse-defaults
    hostname: clickhouse
    # ports:
    #   - "9000:9000"
    #   - "8123:8123"
    #   - "9181:9181"
    command: "ls -la /etc/clickhouse-server"
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config1.xml
      #- ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
      #- ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      ## - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
      #- ./data/clickhouse/:/var/lib/clickhouse/

Which resulted in the following output from the container:

clickhouse-1               | total 124
clickhouse-1               | drwxrwxrwx    1 1000     1000          4096 Jun 22 14:08 .
clickhouse-1               | drwxr-xr-x    1 1000     1000          4096 Jun 22 14:08 ..
clickhouse-1               | drwxrwxrwx    1 root     root          4096 Feb  2 11:46 config.d
clickhouse-1               | -rw-rw-rw-    1 1000     1000         78357 Feb  1 13:43 config.xml
clickhouse-1               | drwxr-xr-x    2 root     root          4096 Jun 21 20:03 config1.xml
clickhouse-1               | drwxrwxrwx    2 root     root          4096 Feb  2 11:46 users.d
clickhouse-1               | -rw-rw-rw-    1 1000     1000          5672 Feb  1 13:43 users.xml

As can be seen, the config1.xml file has been mounted as a directory!!

Docker Engine Info (with docker host running on AlmaLinux 9):

Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:30:53 2024
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 26.1.4
API version: 1.45 (minimum version 1.24)
Go version: go1.21.11
Git commit: de5c9cf
Built: Wed Jun 5 11:29:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.33
GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Docker Engine Info (with docker host running on Flatcar):

Client: Docker Engine - Community
Version: 26.1.4
API version: 1.43 (downgraded from 1.45)
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:30:53 2024
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 24.0.9
API version: 1.43 (minimum version 1.12)
Go version: go1.20.12
Git commit: fca702de7f71362c8d103073c7e4a1d0a467fadd
Built: Tue May 21 11:03:41 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.13
GitCommit: 99b8088b873ba42b788f29ccd0dc26ebb6952f1e
runc:
Version: 1.1.12
GitCommit: 51d5e94601ceffbbd85688df1c928ecccbfa4685
docker-init:
Version: 0.19.0
GitCommit: de40ad007797e0dcd8b7126f27bb87401d224240

[jdall]

FYI, the issue appears to also be present when deploying to a flatcar instance which has not been initialized for docker swarm usage.

The steps:
a. git clone -b main https://github.com/SigNoz/signoz.git
b. cd signoz/deploy/docker
c. docker compose -f clickhouse-setup/docker-compose.yaml up

Results in the following error being shown:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "***/signoz/deploy/docker/clickhouse-setup/clickhouse-config.xml" to rootfs at "/etc/clickhouse-server/config.xml": mount ***/signoz/deploy/docker/clickhouse-setup/clickhouse-config.xml:/etc/clickhouse-server/config.xml (via /proc/self/fd/7), flags: 0x5000, data: context="system_u:object_r:container_file_t:s0:c336,c904": not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

[tormath1]

Hi @jdall, thanks for the details and for the reproduction steps. I can't reproduce yet the issue, can you share those information:

• how did you install docker compose?
• Is the Git repository cloned with a specific user (different from core)
• between your tests, did you purge everything? (docker compose -f clickhouse-setup/docker-compose.yaml down -v) to be sure there is no leftovers from previous tests?

EDIT:
I've done this:

$ git clone -b main https://github.com/SigNoz/signoz.git
$ cd signoz/deploy/docker
$ docker compose -f clickhouse-setup/docker-compose.yaml up clickhouse -d
$ docker compose -f clickhouse-setup/docker-compose.yaml exec clickhouse head /etc/clickhouse-server/config.xml
<?xml version="1.0"?>
<!--
  NOTE: User and query level settings are set up in "users.xml" file.
  If you have accidentally specified user-level settings here, server won't start.
  You can either move the settings to the right place inside "users.xml" file
   or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> here.
-->
<clickhouse>
    <logger>
        <!-- Possible levels [1]:

[jdall]

Thanks, @tormath1

Actually, your question "Is the Git repository cloned with a specific user (different from core)" makes me realize that my problem is probably caused by the fact that I have checked out the code on my workstation, while I'm trying to deploy to a flatcar instance running on a remote host.

So basically, I guess I'm trying to do something that docker compose is not meant to do in the first place :/

Please feel free to close this bug report.

[tormath1]

@jdall ah yes, that's possible indeed. What you could do as a workaround would be to upload the files (clickhouse-config.xml and friends) on Flatcar via initial provisioning (Ignition).
A Systemd unit could take of running the docker compose command.

Similar to this: #894 (comment)

Do not hesitate to join our Matrix or Slack channel if you want to discuss and explore this method with Flatcar users: https://github.com/flatcar/Flatcar?tab=readme-ov-file#chat