Systemd not starting - Failed to connect to bus: No such file or directory

[FedericoAntoniazzi]

Hi everyone, I'm trying to test a role that installs postgresql using the docker driver for molecule but it seems that systemd is not starting.

Symptoms

• After postgres installation I get this log
• Systemctl commands do not connect to bus

root@ubuntu2004:/# systemctl show
Failed to connect to bus: No such file or directory
root@ubuntu2004:/# systemctl status
Failed to connect to bus: No such file or directory
root@ubuntu2004:/# systemctl status postgresql
Failed to connect to bus: No such file or directory

(this container has been created using molecule create && molecule login)

Molecule configuration

---
dependency:
  name: galaxy
driver:
  name: docker
platforms:
  - name: ubuntu2004
    image: docker.io/geerlingguy/docker-ubuntu2004-ansible:latest
    privileged: true
    command: "/lib/systemd/systemd"
    tmpfs:
      - /run
      - /tmp
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    capabilities:
      - SYS_ADMIN
    pre_build_image: true
provisioner:
  name: ansible
verifier:
  name: ansible
lint: |
  set -e
  yamllint .
  ansible-lint .

My setup

• ansible: 2.11.3
• molecule: 3.4.0
• molecule-docker: 0.2.4
• python: 3.9
• docker: 20.10.8
• OS: Archlinux

[maxhoesel]

Fyi, this is an issue between Docker and newer systemd versions (248 and up): systemd/systemd#19245.

One possible workaround is to boot with the kernel parameter systemd.unified_cgroup_hierarchy=0

[FedericoAntoniazzi]

Thank you for pointing this out, I wouldn't have thought the problem was systemd.

[LorenzoBettini]

I'm experiencing this problem as well in Ubuntu 21.10 where systemd is version 248...
I was wondering whether this might be an alternative solution https://github.com/gdraheim/docker-systemctl-replacement

[socialsorcery]

I had a similar issue on MacOS with Docker Desktop (which recently moved to cgroup v2). The only solution I found was to install a beta build mentioned in a github issue and revert back to cgroup v1.

[thom-vend]

If you are using docker for mac workaround can be to go back on cgroup v1

# Stop running Docker
test -z "$(docker ps -q 2>/dev/null)" && osascript -e 'quit app "Docker"'
# Install jq and moreutils so we can merge into the existing json file
brew install jq moreutils
# Add the needed cgroup config to docker settings.json
echo '{"deprecatedCgroupv1": true}' | \
  jq -s '.[0] * .[1]' ~/Library/Group\ Containers/group.com.docker/settings.json - | \
  sponge ~/Library/Group\ Containers/group.com.docker/settings.json
# Restart docker desktop
open --background -a Docker

^ copy pasted from: docker/for-mac#6073 (comment)

[NiftyMist]

I'm having a similar issue running on Pop!_OS 21.04. Trying to see if we can migrate our existing molecule testing pipelines from vagrant/virtualbox setups to a docker based methodology. Using this image I was unable to test a role because it wants to enable a systemd service. But obviously it cannot if systemd is not running. So I decided to create a new virtual environment with fully up to date tools and now I'm getting a completely new error.

My setup

• ansible: 2.12.2
• molecule: 3.6.1
• molecule-docker: 1.1.0
• python: 3.9.7
• docker: 20.10.12
• OS: Pop!_OS 21.04

I'm able to create an instance just fine, but when trying to login (with the same molecule.yml as above) I get the following error:

❯ molecule login
INFO     Running default > login
Traceback (most recent call last):
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/bin/molecule", line 8, in <module>
    sys.exit(main())
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/molecule/command/login.py", line 166, in login
    base.execute_subcommand(scenario.config, subcommand)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/molecule/command/base.py", line 149, in execute_subcommand
    return command(config).execute()
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/molecule/logger.py", line 188, in wrapper
    rt = func(*args, **kwargs)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/molecule/command/login.py", line 101, in execute
    self._get_login(hostname)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/molecule/command/login.py", line 146, in _get_login
    app.runtime.exec(cmd)
  File "/home/dmitchell/ansible/ansible-latest-virtualenv/lib/python3.9/site-packages/ansible_compat/runtime.py", line 138, in exec
    result = run_func(
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/env docker exec -e COLUMNS=236 -e LINES=56 -e TERM=bash -e TERM=xterm -ti ubuntu2004 bash'

I can manually exec into the container. systemd is still not running.

❯ docker exec -it ubuntu2004 /bin/bash
root@ubuntu2004:/# systemctl show
Failed to connect to bus: No such file or directory

[LorenzoBettini]

Fyi, this is an issue between Docker and newer systemd versions (248 and up): systemd/systemd#19245.

One possible workaround is to boot with the kernel parameter systemd.unified_cgroup_hierarchy=0

By booting with that kernel parameter you mean to boot the main OS, right?

[gomex]

I am using Docker for Mac and I am getting the same error. Using this version 4.4.0 (72295) with "deprecatedCgroupv1": true it is working, but the molecule login doesn't work:

...
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/env docker exec -e COLUMNS=136 -e LINES=44 -e TERM=bash -e TERM=xterm -ti ubuntu-20.04-iacroleredis bash'

[FedericoAntoniazzi]

Hi @gomex, This is a molecule problem, the temporary solution is to just copy that command manually and running it separately.

[ned300889]

Hi @gomex, the resolution above is technically true you just need to install the previous version to get this running molecule==3.5.2 works fine. Just uninstall the module then reinstall it with that version number and you're fine.

[geerlingguy]

I've started getting this too, on latest molecule with latest Docker Desktop for Mac :(

[kozicpetar]

Looks like we need to move everything to some kvm like virtualbox or vmware where we have full kvm.

[ned300889]

There are two combined issues, molecule issue is mentioned here and this will be seen throughout providers i believe.

Docker issue is related to systemd not starting as mentioned here

I believe a resolution to molecule is pinning the version within the docker build process for now whilst it gets resolved and for the docker issue put a note on the README about a required desktop docker version for MacOS.

[NiftyMist]

@kozicpetar I've been a fan of using virtualbox provider for molecule testing. Been using it with my compnay for over a year now. Just hard because I don't think something that would work for github actions unless I'm wrong. Github actions seems to be based on containers from what I understand.

[kozicpetar]

@NiftyMist yes, you are in the right. My primary problem is running tests on GHA because support containers.

@geerlingguy Jeff, do you have some idea how we can sort this problem ?

[geerlingguy]

Related? geerlingguy/docker-debian11-ansible#4

[abhilashmandaliya]

Fyi, this is an issue between Docker and newer systemd versions (248 and up): systemd/systemd#19245.

One possible workaround is to boot with the kernel parameter systemd.unified_cgroup_hierarchy=0

You saved my life :-)

[kozicpetar]

I sort problem on my Mac with Docker desktop version Docker Desktop 4.7.1 (77678)

I built new container for Ubuntu and Debian without VOLUME [ "/sys/fs/cgroup" ]

This is Dockerfile for Ubuntu 20.04:

FROM ubuntu:focal

ENV container docker
ENV LC_ALL C
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update \
    && apt-get install -y python3 sudo bash ca-certificates iproute2 python3-apt aptitude systemd systemd-sysv \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN rm -f /lib/systemd/system/multi-user.target.wants/* \
    /etc/systemd/system/*.wants/* \
    /lib/systemd/system/local-fs.target.wants/* \
    /lib/systemd/system/sockets.target.wants/*udev* \
    /lib/systemd/system/sockets.target.wants/*initctl* \
    /lib/systemd/system/sysinit.target.wants/systemd-tmpfiles-setup* \
    /lib/systemd/system/systemd-update-utmp*

RUN systemctl set-default multi-user.target

CMD [ "/lib/systemd/systemd", "log-level=info", "unit=sysinit.target" ]

I do same for Debian 11 and Ubuntu 18.04

My molecule file, you can see there is commented volumes:

platforms:
  - name: Ubuntu-18.04
    image: docker-ubuntu1804-ansible:latest
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    #volumes:
    #  - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true
  - name: Ubuntu-20.04
    image: docker-ubuntu2004-ansible:latest
    #command: "/lib/systemd/systemd"
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    #volumes:
    #  - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true

After this, all my roles with systemd works as expected.
NOTE: I didn't enable deprecated `"deprecatedCgroupv1": true``

[flymia]

@kozicpetar I can confirm that using your Dockerfile I am also able to run systemd specific tasks on Ansible using Molecule on my Mac Mini M1.

[LorenzoBettini]

Starting from the fact that I have no problem when running molecule tests in GitHub Actions (in ubuntu virtual machines), the solution of @kozicpetar still does not work for me: neither in Ubuntu nor in Fedora.

What works for me is the following one (which I was also suggesting in #18 (comment)):

FROM ubuntu:22.04
LABEL maintainer="Lorenzo Bettini"

ENV container docker
ENV LC_ALL C
ENV DEBIAN_FRONTEND noninteractive

# software-properties-common is required for codename in ppa to work automatically
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        software-properties-common \
        python3 sudo bash ca-certificates iproute2 python3-apt \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
    && rm -Rf /usr/share/doc && rm -Rf /usr/share/man

ADD https://raw.githubusercontent.com/gdraheim/docker-systemctl-replacement/master/files/docker/systemctl3.py /usr/bin/systemctl
RUN chmod +x /usr/bin/systemctl

CMD ["/usr/bin/systemctl"]

With this one, my playbooks and tasks that enable or disable services (even at the user level) work in molecule.

[MohammadrezaNasrabadi]

This problem has been occurred for me too.

After two days of troubleshooting, I finally resolve my own issue.

I'm using debian 11 operating system and as docker documentation said, ( this link ) from debian 11 to latest , cgroup v2 is using.

So I guessed that the problem should be from changing in version of cgroup . Because my co-worker that uses ubuntu 20.04 didn't face such problem.

BTW, after so many tries I decided to mount /sys directory to container instead of /sys/fs/cgroup in read only type and my problem has been solved.

Hope it will be helpfull for others.

[atj]

I tried the various solutions listed on this thread and unfortunately none of them worked for me on Arch Linux.

However after a lot of digging, I found that setting the systemd.unified_cgroup_hierarchy=false kernel option resolved the issue. I'm currently running:

• Linux v5.15.50-1-lts
• systemd v251
• docker v20.10.17

Molecule platform configuration:

platforms:
  - name: instance
    image: "geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu2004}-ansible:latest"
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true

$ molecule create
$ molecule login --host instance
root@server:/# systemctl status     
● server
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Tue 2022-06-28 09:12:10 UTC; 2min 4s ago
   CGroup: /docker/0c7f492573a94bf0f24f605e01802fd3bc74b1e7ab7734f709bfeffb621a33ad
           ├─481 bash
           ├─495 systemctl status
[...]

[d-mankowski-synerise]

After upgrading AKS to 1.25 (which also upgraded Ubuntu image on K8s nodes from 18.04 to 22.04, hence upgraded cgroups to v2) we encountered the above issue as well and neither solution from this thread worked. After some extensive googling and thanks to this answer I came up with this:

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

ENV pip_packages "ansible"

# Install dependencies.
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
       apt-utils \
       build-essential \
       locales \
       libffi-dev \
       libssl-dev \
       libyaml-dev \
       python3-dev \
       python3-setuptools \
       python3-pip \
       python3-yaml \
       software-properties-common \
       rsyslog systemd systemd-cron sudo iproute2 \
    && apt-get clean \
    && rm -Rf /var/lib/apt/lists/* \
    && rm -Rf /usr/share/doc && rm -Rf /usr/share/man
RUN sed -i 's/^\($ModLoad imklog\)/#\1/' /etc/rsyslog.conf

# Fix potential UTF-8 errors with ansible-test.
RUN locale-gen en_US.UTF-8

# Install Ansible via Pip.
RUN pip3 install $pip_packages

# Install Ansible inventory file.
RUN mkdir -p /etc/ansible
RUN echo "[local]\nlocalhost ansible_connection=local" > /etc/ansible/hosts

# Remove unnecessary getty and udev targets that result in high CPU usage when using
# multiple containers with Molecule (https://github.com/ansible/molecule/issues/1104)

RUN rm -f /lib/systemd/system/multi-user.target.wants/* \
  /etc/systemd/system/*.wants/* \
  /lib/systemd/system/local-fs.target.wants/* \
  /lib/systemd/system/sockets.target.wants/*udev* \
  /lib/systemd/system/sockets.target.wants/*initctl* \
  /lib/systemd/system/sysinit.target.wants/systemd-tmpfiles-setup* \
  /lib/systemd/system/systemd-update-utmp* \
  /lib/systemd/system/getty.target

CMD [ "/lib/systemd/systemd", "log-level=info", "unit=sysinit.target" ]

And after building the image, run it like this: docker run -itd --privileged --cap-add SYS_ADMIN --security-opt seccomp=unconfined --cgroup-parent=docker.slice --cgroupns private --tmpfs /tmp --tmpfs /run --tmpfs /run/lock <image name>

Note: it requires host OS to support cgroups v2 (you can check it by running stat -fc %T /sys/fs/cgroup/ - v2 should print cgroup2fs , v1 should print tmpfs) and to run Docker 20.10+

And after f**king hours of debugging molecule and browsing its source code, we finally managed to get it working.

platforms:
  - name: instance
    image: <image name>
    pre_build_image: true
    privileged: true
    cgroup_parent: docker.slice
    cgroupns_mode: private
    override_command: false
    capabilities:
      - SYS_ADMIN
    security_opts:
      - seccomp=unconfined
    tmpfs:
      - /run
      - /tmp
      - /run/lock

This crap: override_command: false is not documented anywhere and because of the default Molecule behavior, container is started with sleep as PID 1, which causes the following, when checking systemctl status by ansible: {"changed": false, "msg": "Service is in unknown state", "status": {}}. Using /lib/systemd/systemd as the entrypoint (hence, systemd == pid 1) fixed everything

[AzraelG]

I solved the problem changing the volume configuration from /sys/fs/cgroup:/sys/fs/cgroup:ro to /sys/fs/cgroup:/sys/fs/cgroup:rw.

Molecule platform configuration:

platforms:
  - name: instance
    image: "geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu2004}-ansible:latest"
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
    privileged: true
    pre_build_image: true

[sirkubax]

---
dependency:
  name: galaxy
driver:
  name: docker
platforms:
  - name: instance
    image: "geerlingguy/docker-${MOLECULE_DISTRO:-centos7}-ansible:latest"
    command: /lib/systemd/systemd
    volumes:
    #  - /sys/fs/cgroup:/sys/fs/cgroup:ro
      - /sys/fs/cgroup:/sys/fs/cgroup:rw

    pre_build_image: true
    privileged: true
    capabilities:
      - SYS_ADMIN
    tmpfs:
      - /run
      - /tmp
    cgroupns_mode: host

provisioner:
  name: ansible
  playbooks:
    converge: ${MOLECULE_PLAYBOOK:-converge.yml}