onboot service not persistent across newer firmware upgrades

[andrewmiskell]

Describe the bug
After upgrade from 1.8.0 to 1.8.2-10 (and also from 1.8.2-10 to 1.8.3-2) the onboot service was wiped and not reloaded

To Reproduce
Steps to reproduce the behavior:

1. Have onboot scripts installed
2. Upgrade firmware from 1.8.0 to newer revision
3. Check onboot scripts after firmware upgrade

Expected behavior
Onboot scripts are reinstalled from dpkg cache

UDM Information

• Variant - Dream Machine Base
• Firmware Version: 1.8.0

Additional context
There very well may be nothing that could be done and it might just be a difference on how the firmware upgrades are installed/handled now by the UDM. I just wanted to log it so if nothing can be done, documentation can be updated to reflect that it's no longer persistent.

[dsully]

I'm seeing this as well.

[MarkBurchard]

Also seeing this on my UDMB, after upgrade from 1.8.2-8 > 1.8.3-2 and again on upgrade from 1.8.3-2 > 1.8.3-3.

[MarkBurchard]

Oddly enough, I just installed the 6.0.3.7 Controller release and after a reboot, multicast-relay started automatically. Every reboot since the 1.8.3-2 upgrade, I had to start that manually. No idea why on-boot unbroke, but can't complain.

[Djelibeybi]

I just saw this upgrading to 1.8.3-4.

[pokemon81]

upgrading to 1.8.3-4 same Problem had to start manuel

[apoc4lyps]

I noticed that onboot.d failed during the first boot after an update. subsequent reboots will trigger the onboot.d scripts

[Murgeye]

I noticed that onboot.d failed during the first boot after an update. subsequent reboots will trigger the onboot.d scripts

Same here. It always seems to fail on the first boot after an upgrade, but works after that. Other user added systemd units seem to run fine.

[mlankamp]

I noticed that onboot.d failed during the first boot after an update. subsequent reboots will trigger the onboot.d scripts

Same here. Always have to do a manual reboot after the upgrade, then the podman containers can start correctly

[Murgeye]

@mlankamp you can also always restart the service using systemctl start udm-boot in the unifi-os shell as a workaround.

[Djelibeybi]

Confirmed that another reboot works here too.

[ravens]

I observed that behavior only with the 1.0.2 udm-boot package. The 1.0.1 works fine - I got two routers and I was surprised to have to reboot the second one to fix the upgrade. I suspect the postinst changes are the culprit (original commit between 1.0.1 and 1.0.2 was 282b9bd#diff-60f00db60b712271f12f9fbaadee09d956229745493e2f812a701c28386cf54e, if I am not mistaken).

[SamErde]

I just upgraded to 1.8.3 official with 6.0.41 controller, and can confirm the same behavior. Doesn't start automatically on the first boot after an upgrade, but does on following restarts.

[mojo333]

I observed that behavior only with the 1.0.2 udm-boot package. The 1.0.1 works fine

Just to confirm this, I downgraded to the 1.0.1 version of the package. Worked perfectly on the latest UDM update - no need to manually reboot which is great news - thanks @ravens for the tip.

[leachbj]

Failed to restart for me too; I found this in journalctl after the upgrade.

Dec 19 13:47:34 ubnt ubnt-dpkg-restore[26]: /usr/sbin/policy-rc.d returned 101, not running 'enable udm-boot'
Dec 19 13:47:34 ubnt ubnt-dpkg-restore[26]: /usr/sbin/policy-rc.d returned 101, not running 'start udm-boot'

/usr/sbin/policy-rc.d doesn't exist after the install completed so it seems like part of the update process is disabling the service configuration.

[boostchicken]

Can someone sum up the current state here for me?

From what I see

• 1.0.1 works
• 1.0.2 does not work at all
• The first reboot after an upgrade it does not execute for some reason?

[leachbj]

I think the change @ravens id'ed above looks relevant that policy-rc.d message I saw is probably related to deb-systemd-invoke enable udm-boot vs the direct systemd call you had.

[andrewmiskell]

Can someone sum up the current state here for me?

From what I see

• 1.0.1 works
• 1.0.2 does not work at all
• The first reboot after an upgrade it does not execute for some reason?

1.0.1 works without any issues, even across firmware upgrades
1.0.2 requires an extra reboot after upgrade to start working again, it doesn't reinstall directly after a firmware upgrade, after the second reboot, everything works fine until the next firmware upgrade

[boostchicken]

@spali can you investigate why your 1.0.2 behaves so differently. Its much more complex than mine but hits all the same bases

If we can't then we have to rollback to 1.0.1

[boostchicken]

I think the change @ravens id'ed above looks relevant that policy-rc.d message I saw is probably related to deb-systemd-invoke enable udm-boot vs the direct systemd call you had.

Man, that has to be it right? There is so little that really changed, just the more "debian" way of doing it. Maybe my simpleton way cut through some garbage, i'll see what @spali says and we go from there

[boostchicken]

Just went to 1.8.4 final with 1.0.1 with no issues

[leachbj]

@boostchicken btw this was the link I found when I looked at the problem; https://jpetazzo.github.io/2013/10/06/policy-rc-d-do-not-start-services-automatically/

[spali]

@spali can you investigate why your 1.0.2 behaves so differently. Its much more complex than mine but hits all the same bases

If we can't then we have to rollback to 1.0.1

To be honest, I have no idea why it behaves so differently. But I'm fine with rollback the calls to direct systemd calls. Maybe this is the same problem I observe with #46. Which basically uses the same calls.

Bit of theory: I'm not a debian packages specialist, but during my research on this calls... I did understand, that the deb calls should just postpone the calls to be called after all packages are installed, to have a more control over them if you install more complex stuff. So as long we do not interfere much with other services I think it would be save to revert this calls.
Especially if you all observe this only since 1.0.2.

[aessing]

Hi,

yesterday I updated from 1.8.3 to 1.8.4 firmware on the UDM Pro.

I had onBoot 1.0.2 installed and everything worked like a charm... even after the upgrade, the onBoot works without doing something. I didn't had to do additional boots.

So, no issues with 1.0.2 and 1.8.4.

Cheers
Andre

[darizotas]

Upgrading from 1.8.0 to 1.8.4
udm-boot 1.0.2

I can confirm that I had to reboot a second time to get the container and service running again.

Cheers,
Darío.

[vvuk]

Is there a new udm-boot package with the potential fix, or do I need to build my own? Happy to install and see how it goes -- I've had no on-boot scripts run the past two automatic upgrades. (upgraded to 1.8.4 automatically last night, no on-boot)

[spali]

Not yet, I assume John will merge the fix and build it soon. If you want you can for sure build it by yourself based on the PR.

[JKomoroski]

I suggest we provide explicit instructions for how to update the deb when a release is cut.

[spali]

@JKomoroski
this is already documented in https://github.com/boostchicken/udm-utilities/tree/master/on-boot-script#steps

[smegoff]

FYI - I exhibit this behaviour too.
I have only recently installed pihole on a UDM Pro, and the last couple of firmware updates I am unable to access the web interface for the pihole.
I checked using "podman ps" in SSH and the container isnt running. The quickest way to get the container to run is reboot the device.

[spali]

@smegoff

I checked using "podman ps" in SSH and the container isnt running. The quickest way to get the container to run is reboot the device.

In my case and I tink in any case of this issue, usually unifi-os restart is enough to fix it.

[spali]

BTW: just updated to 1.8.5 and at least with my custom build of #50 it worked as expected and started automatically everything again after the update. So I assume #86 would work too (basically the same fix)

[rdoram]

@spali, I used your v1.1.0 package (udm-boot) and installed two piHole containers and a ntopng container on UDM-base v1.8.5. I'll report back when I update to the next beta/alpha firmware on the UDM to confirm it survived the upgrade. As of now, it's surviving UDM reboots, so I'm optimistic it'll work. Here's the instructions for anyone that wants to run the same test with NTOPNG-UDM. It avoids some of the more complicated networking setup involved with the piHole. Feel free to correct/comment on any of this, I'm a GitHub and Linux noob, so I'm sure there's probably room for improvement :).

Step 1: Install the udm-utilities package...SSH into the UDM:

unifi-os shell

In the UniFi-OS shell, download the package, install it, and go back to the UDM

curl -L https://github.com/spali/udm-utilities/releases/download/v1.1.0-poc-4/udm-boot_1.1.0_all.deb -o /tmp/udm-boot_1.1.0_all.deb
dpkg -i /tmp/udm-boot_1.1.0_all.deb
exit

Step 2: Install ntopng-udm container

Create persistent folders and files for ntopng-udm

mkdir -p /mnt/data/udm-boot/data/ntopng/redis
mkdir -p /mnt/data/udm-boot/data/ntopng/lib
touch /mnt/data/udm-boot/data/ntopng/GeoIP.conf
curl -Lo /mnt/data/udm-boot/data/ntopng/ntopng.conf https://github.com/tusc/ntopng-udm/blob/master/ntopng/ntopng.conf?raw=true
curl -Lo /mnt/data/udm-boot/data/ntopng/redis.conf https://github.com/tusc/ntopng-udm/blob/master/ntopng/redis.conf?raw=true

Edit the file at /mnt/data/udm-boot/data/ntopng/ntopng.conf to determine which interfaces NTOP will analyze.

Enter udm-boot bash shell:

podman exec -it udm-boot /bin/bash

Use podman to create container

podman create \
    --restart always \
    --net=host \
    --name ntopng \
    -e TZ="America/Chicago" \
    -e VIRTUAL_HOST="ntopng" \
    -e PROXY_LOCATION="ntopng" \
    -e IPv6="False" \
    -v /mnt/data/udm-boot/data/ntopng/GeoIP.conf:/etc/GeoIP.conf \
    -v /mnt/data/udm-boot/data/ntopng/ntopng.conf:/etc/ntopng/ntopng.conf \
    -v /mnt/data/udm-boot/data/ntopng/redis.conf:/etc/redis/redis.conf \
    -v /mnt/data/udm-boot/data/ntopng/lib:/var/lib/ntopng \
    --hostname ntopng \
    docker.io/tusc/ntopng-udm:latest

Configure systemd to load containers at startup

podman generate systemd ntopng >/etc/systemd/system/ntopng.service
systemctl daemon-reload
systemctl enable ntopng.service
systemctl start ntopng.service

[leachbj]

I did a manual build for the Debian package, installed that and then did the 1.8.3 to 1.8.5 upgrade and everything came up correctly after the upgrade, no additional restart required.

[boostchicken]

Closing as this is now fixed with a new release!