mosquitto install: fails to start the service on boot

[davidhq]

Creating a bug report/issue:

Required Information:

• DietPi version | cat /DietPi/dietpi/.version

#!/bin/bash
G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=17
G_DIETPI_VERSION_RC=12
G_GITBRANCH=master
G_GITOWNER=Fourdee

• Distro version | echo $G_DISTRO_NAME or cat /etc/debian_version
  9.4

• Kernel version | uname -a

Linux lab 4.14.71-v7+ #1145 SMP Fri Sep 21 15:38:35 BST 2018 armv7l GNU/Linux

• SBC device | echo $G_HW_MODEL_DESCRIPTION or (EG: RPi3)
  RPi 3 Model B (armv7l)

Steps to reproduce:

1. sudo apt-get install mosquitto
2. reboot ->> PROBLEM -> mosquitto not running:
   sudo systemctl status mosquitto
   Active: active (exited)

SOLUTION (strange!):

I tried everything, nothing helped, then by coincidence I issues:

sudo systemctl disable mosquitto

rebooted, and everything works as expected!

mosquitto uses autogenerated service file, not sure where the confusion is and why systemctl disable actually fixes the problem instead of completely disabling the loading of service on boot...

I now have a functioning mqtt broker installation but this is weird... please check.. and I hope this report helps others in the meantime...

I tried this on two different devices, the problem is reproducible...

root@lab:~$ /usr/sbin/mosquitto -v
1542224183: mosquitto version 1.4.10 (build date Wed, 17 Oct 2018 19:03:03 +0200) starting

[MichaIng]

@davidhq
Thanks for your report. Will check and try to replicate.

The service should be disabled by DietPi automatically (to leave control to dietpi-services).

---

[ SUB1 ] DietPi-Services > dietpi_controlled
[  OK  ] DietPi-Services | dietpi_controlled : mosquitto
...

 DietPi-Software
─────────────────────────────────────────────────────
 Mode: Installation completed

[ SUB1 ] DietPi-Services > start
[  OK  ] DietPi-Services | start : mosquitto
[  OK  ] DietPi-Services | start : cron
[ SUB2 ] DietPi-Process_tool > Apply
[  OK  ] DietPi-Process_tool | Completed
root@VM-Stretch:~# systemctl status mosquitto
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
...

• Hmm, dietpi_controlled : mosquitto indicates service being disabled, but the status gives no indication about this. Perhaps this is the case for all init.d services.
  ...

root@VM-Stretch:~# systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
...

• generated (mosquitto) vs enabled (networking)

root@VM-Stretch:~# systemctl disable networking
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
root@VM-Stretch:~# systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; disabled; vendor preset: enabled)
...

• Should be disabled?

root@VM-Stretch:~# systemctl disable mosquitto
mosquitto.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mosquitto
root@VM-Stretch:~# systemctl status mosquitto
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
...

• 🈯️ Reboot afterwards and service startup went fine. Will retry without manually disabling.
• 🈯️ Went fine as well... But did dietpi-software install 123 which includes service startup and manual reboot. Will retry with automated reboot, excluding prior service startup.
• 🈯️ Went fine as well... Cannot replicate on VM.
  • Service status btw. remains at generated, no idea what this means compared to disabled, but does not cause any issue here.

@Fourdee
Do you have a chance to test on RPi?

@davidhq

I tried this on two different devices, the problem is reproducible...

Was the second device an RPi as well?

Checked our install code. Wow, in case of Mosquitto it is really only apt-get install mosquitto, no manual changes, besides leaving service control to dietpi-services, which mainly delays service startup to end of boot process in reasonable order with other backend processes 😄.

[MichaIng]

Further checking of service status:

root@VM-Stretch:~# systemctl enable mosquitto
mosquitto.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable mosquitto
root@VM-Stretch:~# systemctl status mosquitto
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)

• Enabling service does not change status indication.
• Perhaps declarations are similar to systemd static units?

### BEGIN INIT INFO
# Provides:             mosquitto
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6

• Default-Start is set, but since it's "default", it should not override manual change?
• However it was definitely started by dietpi-services:

Nov 15 12:49:07 VM-Stretch systemd[1]: Startup finished in 2.269s (kernel) + 3.534s (userspace) = 5.803s.
Nov 15 12:49:07 VM-Stretch systemd[1]: Starting Network Time Synchronization...
Nov 15 12:49:07 VM-Stretch systemd[1]: Started Network Time Synchronization.
Nov 15 12:49:07 VM-Stretch systemd[1]: Reached target System Time Synchronized.
Nov 15 12:49:07 VM-Stretch systemd-timesyncd[775]: Synchronized to time server <my_gateway_ip>:123 (<my_gateway_ip>).
Nov 15 12:49:07 VM-Stretch systemd[1]: Stopping Network Time Synchronization...
Nov 15 12:49:07 VM-Stretch systemd[1]: Stopped Network Time Synchronization.
Nov 15 12:49:08 VM-Stretch systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker...
Nov 15 12:49:08 VM-Stretch mosquitto[893]: Starting network daemon:: mosquitto.
Nov 15 12:49:08 VM-Stretch systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.
Nov 15 12:49:08 VM-Stretch systemd[1]: Started Regular background program processing daemon.
Nov 15 12:49:08 VM-Stretch cron[900]: (CRON) INFO (pidfile fd = 3)
Nov 15 12:49:08 VM-Stretch cron[900]: (CRON) INFO (Running @reboot jobs)
Nov 15 12:49:12 VM-Stretch dropbear[1077]: Child connection from <my_client_ip>:50403

[Fourdee]

RPi test, install + reboot:

• 🈯️

root@DietPi:~# dpkg -l | grep syslog
root@DietPi:~# systemctl status mosquitto.service -l
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
   Active: active (running) since Fri 2018-11-16 16:51:21 GMT; 52s ag
o
     Docs: man:systemd-sysv-generator(8)
  Process: 936 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCC
ESS)
      CPU: 57ms
   CGroup: /system.slice/mosquitto.service
           └─942 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Nov 16 16:51:21 DietPi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message bro
ker...
Nov 16 16:51:21 DietPi mosquitto[936]: Starting network daemon:: mosquitto.
Nov 16 16:51:21 DietPi systemd[1]: Started LSB: mosquitto MQTT v3.1 message brok
er.

[davidhq]

Not sure what these reports mean -) Anyway have you reproduced the problem? Check that process is actually running even if systemctl reports it’s active.. actually it said Active (exited) and it wasn’t reachable (running) when I was trying.. I solved the issue in this strange way as described..

…

On 16 Nov 2018, at 17:53, Dan ***@***.***> wrote: RPi test: 🈯️ ***@***.***:~# dpkg -l | grep syslog ***@***.***:~# systemctl status mosquitto.service -l ● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled) Active: active (running) since Fri 2018-11-16 16:51:21 GMT; 52s ag o Docs: man:systemd-sysv-generator(8) Process: 936 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCC ESS) CPU: 57ms CGroup: /system.slice/mosquitto.service └─942 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf Nov 16 16:51:21 DietPi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message bro ker... Nov 16 16:51:21 DietPi mosquitto[936]: Starting network daemon:: mosquitto. Nov 16 16:51:21 DietPi systemd[1]: Started LSB: mosquitto MQTT v3.1 message brok er. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Fourdee]

@davidhq

Hi David,

Yep, the service and process are running (checked htop).

The only thing I can suggest is we create a systemd service for Mosquitto, and, remove the init.d version. This should prevent any issues with dietpi-services disabling the service (which we are unable to replicate).

Either way, systemd service is preferred.

---

Notes

$(which mosquitto) -c /etc/mosquitto/mosquitto.conf

[MichaIng]

€: Lol double answer.
@Fourdee
Maybe we can check current upstream mosquitto for their systemd service and/or if not there as well, commit one. But then it's up to the Debian maintainers to implement and no chance to have this in Stretch anymore in case. Simple test as well to check APT package on Buster, perhaps it's already systemd there.

@davidhq
Nope, we were not able to reproduce. I just found an interesting state report of the service but it was up and running after reboot during my VM test as well as on Fourdees RPi test.

So not sure what happened on your case. Which other software have you installed besides mosquitto?

[Fourdee]

@MichaIng @davidhq

I've updated the service to systemd regardless. init.d script was around 200 lines just to run $(which mosquitto) -c /etc/mosquitto/mosquitto.conf

Install test + service + process 🈯️

[davidhq]

I’ll dedicate more time to this again soon, I’m not at thome, for now just: - haven’t installed much other software , at least not something very related to mosquitto or anything that would interfere with systemd and such - I intalled mosquitto via apt-get not dietpi-software— not sure if it matters but I think I forgot to mention this fact - great if you simplified things now, this is a great step no matter what the real issue is.. so if init.d is out now, I think and hope this might resolve the problem.. but still please try one more time to reproduce with the old way, just try to install mosquitto manually.. and id ou can, pleas “enlighten me” whether in general there is any difference between two ways of installing Thank you!

…

On 16 Nov 2018, at 20:05, Dan ***@***.***> wrote: @MichaIng @davidhq I've updated the service to systemd regardless. init.d script was around 200 lines just to run $(which mosquitto) -c /etc/mosquitto/mosquitto.conf — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MichaIng]

@davidhq
Nope, in case of mosquitto there is absolutely no difference in apt and dietpi software install.

[MichaIng]

@davidhq
Since you could easily resolve it on your system and we are not able to replicate it, I guess we can close this topic?
Feel free to reopen, if required and we keep it in mind, if any other user reports similarly, to then investigate deeper.

[davidhq]

Ok, I agree... and as you said, you simplified something so I look forward to not having this issue with a new release :) If I do, I'll write here.. thank you so far!

[g3ntleman]

I had the same problem after installing the current mosquitto fresh via apt:

"/etc/init.d/mosquitto status" showed server boot-up messages, but a portscan did not reveal any open ports. "mosquitto -d" made the service available, though not boot-safe.

Unfortunately, I did both, the suggested

• "sudo systemctl disable mosquitto" and an
• "apt remove" and install via "dietpi-software"
  without checking in-between.

Now my mosquitto is finally reboot-safe.

BTW: Keep up the good work on diet-pi. I love it over any other comparable Linux distro!

Info:
G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=22
G_DIETPI_VERSION_RC=3
G_GITBRANCH=master
G_GITOWNER=Fourdee

Hardware: RPi B (armv6l)

Working version (vom dietpi_software):
mosquitto version 1.4.10 (build date Wed, 13 Feb 2019 00:45:38 +0000)

[MichaIng]

@g3ntleman
Actually we replace the sysvinit service (/etc/init.d/mosquitto) with a systemd unit: /etc/systemd/system/mosquitto.service
EDIT: Ah lol this was done because of this issue actually 😉.
Did you install mosquitto via APT first as well or via dietpi-software initially? In case the info below is obsolete 😄.

---

With /etc/apt/apt.conf.d/99-dietpi-forceconf APT should not re-create manually removed files. It it does, this might cause an issue if the APT package scripts enable/start the service in a special way:

• systemctl enable/start mosquitto and service mosquitto enable/start should be no issue, since systemd ignores sysvinit services if a systemd unit is present.
• But /etc/init.d/mosquitto start and the sysvinit specific service enabling could lead to both services running concurrently.

I will verify if/how the APT install handles things.
🈯️ Tested reinstall, upgrade and downgrade through, the sysvinit service is never re-created (even without the mentioned APT config it does not even ask to) and the systemd service is enabled instead. So that should be fine.

[AruljothySundaramoorthy]

Still I'm facing the same issue, while installing mosquitto

systemd[1]: /etc/systemd/system/mosquitto.service:17: Missing '='.

[MichaIng]

@Arruls8
This is not the same issue. The service file we create definitly does not miss a =: https://github.com/MichaIng/DietPi/blob/master/dietpi/dietpi-software#L11108-L11122
Actually it even only has 15 lines.

Could you paste the output of the following command:

cat /etc/systemd/system/mosquitto.service /etc/systemd/system/mosquitto.service.d/*

Probably a drop-in config is the issue.