[Discussion] Better service management

[spali]

I made an issue for discussion what started in #45.

I would like to implement a clean service management instead of just the current on_boot.sh one shot script that executes other scripts, which also supports scheduling.

First idea was to use systemd, supervisor or some other "process manager" inside a container to start the stuff that is currently in on_boot.d. We could also reuse the systemd service from the unifi-os container.
But both solutions have the same problem. podman is only available on the host, so we need to ssh from inside the container to the host to start containers.
Ubiquiti has already solved part of the problem by using an ssh_proxy command, which can be used to execute commands on the host (UDM) outside the unifi-os container. But then we need to prefix every command execution in the service files with this ssh proxy.

So the challenge would be to make this transparent for the service definition (unit files in case of systemd).

One possible compromise could be to only "wrap" the podman command inside the unifi-os container to a shell script that use the ssh_proxy and pass everything to the podman on the host. And everything else (script's etc.) would be executed inside the unifi-os container.

Another possibility would be to somehow run our own podman including systemd inside a udm-boot container. Not quit sure how easy this is to implement. But would have the advantage of beeing more isolated to not mess to much with the UDM host and the unifi-os container.

Any other ideas, improvements or comments?

[spali]

Hi

@boostchicken Just a progress report.

I got a working container which contains systemd and nested podman running.
From there on we have a full blown debian with podman we can work with.

The idea would be to only start this container by the systemd of unifi-os through the ssh_proxy.
The data directory can be mounted into it depending on if the external drive is there or not. I would suggest to use the external drive if available, otherwise the internal flash.
If we also mount the nested podman's directories to the drive we do not clutter the podman storage of the UDM itself.
Even we don't have an external drive, I was worried to fill /mnt/data/podman/storage which is used by UDM.
This way we have the control over the data can also cleanup by ourself without conflicting with the unifi data. The only data that would be there is the one single (let's call it UDM-BOOT) container and the image of it. But the content of this container and all nested ones, can be controlled by us.

I'm working on packaging this test into a deb package, so we could test it a bit more. Especially the network part is not tested yet. Just have the premise this works somehow if we use the host-network.

[spali]

just as reference: #50

[boostchicken]

@spali been super busy lately, your PR is huge, let me give it a look. try to this weekend

[spali]

@boostchicken
Don't worry. It's also a kind of a breaking change.. maybe some doc adaption is missing.

In short it does completely separate the podman and the custom containers by running it in a udm-boot container.
With that we have the latest and greatest podman version available and some other limits removed.

Also before merging I would like to improve the podman network part.
But the network part is not my strength.
I would like to see a solution where the user doesn't have to edit some files to get the basic use-cases working, like setting the VLAN and IP in the dns scripts.

The simplest solution would be to configure the default network as host to prevent podman from creating iptables entries by default to prevent issues like #49. Maybe we should do this anyway independent of the other network stuff for users just want to run a simple service container like ntp.

First idea was, to implement a macvlan with IPAM dhcp to allow the containers getting their IP dynamically from the UDM itself. But because macvlan can't talk to the host itself this doesn't work without an external DHCP.
There is a workaround to make this kind of working with a second macvlan that act's as routing interface to the host. But this requires to setup again an IP and routes which is kind of the same as the current dns scripts.

Second try was a bridge interface with ipam set to dhcp and the cni dhcp daemon (daemon itself already included in this PR).
This works, but without respecting the --mac-address option during container creation. So it get's always a random mac.
see containers/podman#1136 (comment) and containernetworking/plugins#450
Don't know if it would require changes in the cni dhcp daemon to use always the containers defined mac address.

If we can get a bridged network working using a fixed mac address per container that works with dhcp, this would simplify the networking part a lot for users that just want to run containers with an dedicated IP for the dns stuff.

[senseisimple]

This is exciting development!
I don't see a reason why the two branches in development https://github.com/boostchicken/udm-utilities/tree/package-manager/udm-launcher and https://github.com/spali/udm-utilities/tree/wip_container/on-boot-script couldn't complement each other... referring to PR #50

udm-launcher can help to manage the udm-boot container system, creating a wrapper around the various commands required to create/start/stop and debug the custom container images running within udm-boot.

In fact if you combine the two it makes for a whole user friendly package, then instruction/references could simply go like udm-launcher install https://github.com/....<raw script with install/create instructions> to install in the udm-boot container and udm-launcher install <native docker/podman images> <podman create args>

Also I can see use in having both the "legacy" boot script on the baremetal UDM and the systemd based (in the udm-boot container) in place, similar to how systemd still works with sysv init.d.. one is much more familiar and simple to use, but more importantly allows the use of both the native podman (or other boot scripts) and a more advanced mechanism in the udm-boot container.

With that said, I'd love to help out though there hasn't been any activity in 3 months on this so I would hate to start making changes or coming up with any ideas to either branch if additional progress has been made behind the scenes by either of you

[spali]

@senseisimple
Hi
There is currently no progress because I didn't had time to dig more into. If you wanna add something to this PR, don't worry. Currently not actively working on it. My branch is still up to date with the last progress.

Currently I have two things, that I would like to have solved.

• After the last firmware upgrade, the container didn't start automatically without a reboot. Not sure if this is a bug I introduced in my branch. But a simple restart of the unifi-os container did work and started the udm-boot container. Just didn't had time to investigate it.
• I would like to somehow make the networking easier. I want to have the network pre-configured with a default one, that doesn't have the possibility to place iptables rules that do break unifi-os. I think masquarding or prerouting rules created by podman in the default network configuration is something that unifi-os does not like... check some of the issues in this repo. But iptables and podman networking is not my strength ;) My first plan was to create a macvlan which should least bother with the iptable rules and can have full network access. But because of the limitation, that a macvlan can't talk to the host (UDM in this case) it doesn't work as expected if i.e. you wan't to use DHCP with the UDM itself.

So if you have ideas or time to help here out, let me know.

[boostchicken]

@spali
What I wanted to do before my life got all crazy was more a docker-compose or podman-compose implementation to the UDMP. There are some out there written in Python I believe that could very easily
There is also this: https://github.com/containers/podman-compose

That only solves half of the problem though. That is where the abandoned udm-launcher comes in. I hate to make yet another package manager, but short of stealing deb packages and moving them back up to the main OS from the unifi-os container, there is nothing.

@senseisimple @spali I leave this to you guys, I might have time to pitch in here shortly. If a decision is made on how we are going to move forward, I will support it.

I still like the idea of the udm-launcher app that can just read config files and apply networking and file changes. Makes it dead simple.

[spali]

In my eyes a basic podman knowledge should be expected from the users. But I think the UDM specific stuff can be made easier for the users. Also the more advanced stuff like networking can be abstracted maybe.
I still wish we could somehow have a macvlan as default network with dhcp to the host (UDM). But still have no idea how to implement it without to much knowledge about the users configuration like subnet etc.

[peacey]

I would like to add my opinion that I would really like the option to revert to the old non-container way without installing the boot container. I already have a custom container I use to run all my custom services that I created with my own parameters. I do not need another one that I did not create.

I prefer not to excessively use docker and have lots of containers on my system, so I like to put everything in one custom container. This boot script gave me an easy solution to start my custom container on boot, or even do other custom non-container setups like modifying system configuration files on startup, or running compiled golang programs directly without a container. If you force us to install the udm-boot container, then I will have an extra container installed just wasting space and resources.

If the non-container way works properly without issue, I would really appreciate the option to not install the udm-boot container and just use the old way. It was versatile, could be adapted for anything, and worked very well for many use cases, not just container startups.

I appreciate the work you put into all this, but I just prefer the option because the new way doesn't fit all my use cases or makes solving them unnecessarily complex.

[spali]

@peacey
I understand your point.
The big advantage is that it's a lot more modularized. What usually is the point of containerization. Instead of one big container containing everything. But it's a feasible point of you anyway.

Technical:
Current implementation starts the on_boot.sh from within the udm-boot container as a service.
But it should be possible to move the start of this again back to the unifi-os container as it's own service.
Then changing the container service to create the container on the fly and not in advanced.
And provide a way to disable this service.
This would more or less the 1.0.x variant with optional udm-container.

Or just live with the container, as it does almost not need any resources when running idle. Except from a bit of space.

Just to say, it would be do-able, but in my eyes breaks the modularized approach that it should work for anyone a bit.
In my eyes you do exactly the same as my PR, but just for yourself with your customized container ;)

At the end John should and has to decide, he has still to decide anyway how to proceed with this PR in general.
@boostchicken PS: don't feel stressed because of me ;)

I think it's a much cleaner way to have the modularized approach to completely separate and encapsulate everything in a own container to keep the UDM itself as clean as possible, especially for services not just scripts to hack around on the udm.
And I think this repo has already overcome this point of just scripts with all the example containers.

For me, the motivation is, I only have one UDM and it's in production. So I want a "kill-switch" for everything that just works, but with the maximum of flexibility.
This way, you can treat the udm-boot container as a server and not the udm itself, which makes deployment on it much more generic and portable. I.e. I use ansible in my homelab and want to use it too for provision stuff on the udm.

Worst case for me would be, I split of my variant completely which would be also a lot less work on my side ;)

Just to be clear I don't feel offended and I really understand your point, just wanted to explain a bit the background from my view.