tailscaled.service cannot start due to incorrect path

[jmacdonagh]

Description

The tailscale sysext uses the vendor provided tailscaled.service which has ExecStart=/usr/sbin/tailscaled ..., but the binaries are "installed" to /usr/local/sbin. When attempting to start the service it fails to start with the error Job for tailscaled.service failed because of unavailable resources or another system error.

Impact

Tailscale sysext is non-functional.

Environment and steps to reproduce

1. Set-up: Provision flatcar with the following butane config

variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /opt/extensions/tailscale/tailscale-1.76.6-x86-64.raw
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/tailscale-1.76.6-x86-64.raw
    - path: /etc/sysupdate.d/noop.conf
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
    - path: /etc/sysupdate.tailscale.d/tailscale.conf
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/tailscale.conf
  links:
    - target: /opt/extensions/tailscale/tailscale-1.76.6-x86-64.raw
      path: /etc/extensions/tailscale.raw
      hard: false
systemd:
  units:
    - name: systemd-sysupdate.timer
      enabled: true
    - name: systemd-sysupdate.service
      dropins:
        - name: tailscale.conf
          contents: |
            [Service]
            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C tailscale update
        - name: sysext.conf
          contents: |
            [Service]
            ExecStartPost=systemctl restart systemd-sysext

3. Task: Attempt to start service: sudo systemctl start tailscaled.service
4. Action(s): N/A
5. Error:

$ sudo systemctl start tailscaled.service
Job for tailscaled.service failed because of unavailable resources or another system error.
See "systemctl status tailscaled.service" and "journalctl -xeu tailscaled.service" for details.

Expected behavior

Tailscale service should start

Additional information

$ grep ExecStart /usr/local/lib/systemd/system/tailscaled.service
ExecStart=/usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock
$ stat /usr/sbin/tailscaled
stat: cannot statx '/usr/sbin/tailscaled': No such file or directory
$ stat /usr/local/sbin/tailscaled 
  File: /usr/local/sbin/tailscaled
  Size: 35498692  	Blocks: 69334      IO Block: 1024   regular file
...

Thanks!

[jmacdonagh]

The Job for tailscaled.service failed because of unavailable resources or another system error. message is because /etc/default/tailscaled doesn't exist by default for EnvironmentFile.

However, if you sudo touch /etc/default/tailscaled and try to start it still fails (just with a slightly better error message):

$ sudo systemctl start tailscaled.service
Job for tailscaled.service failed because the control process exited with error code.
See "systemctl status tailscaled.service" and "journalctl -xeu tailscaled.service" for details.

Both of these issues (incorrect path, missing EnvironmentFIle) can be fixed with a dropin override:

$ sudo systemctl edit tailscaled.service
...
### Editing /etc/systemd/system/tailscaled.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Service]
EnvironmentFile=
EnvironmentFile=-/etc/default/tailscaled
ExecStart=
ExecStart=/usr/local/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock
ExecStopPost=
ExecStopPost=/usr/local/sbin/tailscaled --cleanup

### Edits below this comment will be discarded
...
$ sudo systemctl daemon-reload
$ sudo systemctl start tailscaled.service

[tormath1]

Hi @jmacdonagh and thanks for the report. Would you be interested to send a pull request to fix this issue?
We could do what you suggested:

• something like: sed -i 's#EnvironmentFile=/etc/default/tailscaled#EnvironmentFile=-/etc/default/tailscaled#g' "${SYSEXTNAME}/usr/local/lib/systemd/system/tailscaled.service"
• update the tailscale{,d} paths to match the one in the unit file:

  sysext-bakery/create_tailscale_sysext.sh

  Lines 37 to 40 in 6cb91d8

  	mkdir -p "${SYSEXTNAME}"/usr/local/{bin,sbin,lib/{systemd/system,extension-release.d}}
  	mv "${TMP_DIR}/tailscale" "${SYSEXTNAME}/usr/local/bin/tailscale"
  	mv "${TMP_DIR}/tailscaled" "${SYSEXTNAME}/usr/local/sbin/tailscaled"

• while at here: bump the released version of Tailscale to 1.17.1

And sorry for the delay, the team is a bit reduced during the festive days.

[jmacdonagh]

Sure, can do.

Looking at the upstream Tailscale release the tar contains tailscale_1.78.1_amd64/systemd/tailscaled.defaults which I believe is supposed to be placed as /etc/default/tailscaled. I will keep the EnvironmentFile= field as is and just have create_tailscale_sysext.sh copy that in.

No worries about the delay, I only found this while trying out Flatcar while on a holiday break :)

[tormath1]

@jmacdonagh great findings! Keep in mind that sysext images are mounted on /usr and /opt only. If we want to ship a configuration file under /etc we need to symlink it. Example:

sysext-bakery/create_nvidia_runtime_sysext.sh

Line 83 in 6cb91d8

C /etc/containerd/config.toml - - - - /usr/share/flatcar/etc/containerd/config.toml

[jmacdonagh]

Here's the PR: #106

I added one comment/question that I could use your feedback on, because I'm fairly new to systemd.