DietPi-Software | ADS-B Feeder: track airplanes using SDRs and feed the data to ADS-B aggregators

[dirkhh]

DietPi-Software | add ADSB-Feeder app that allows feeding ADSB data from one (or more) SDRs to various aggregators (based on https://adsb.im)

Please consider this a draft - I would love feedback on all aspects of this. Did I pick the right category? Any issues with the code? I tried to follow the guidelines I could fine and the comments that you made in our discussions today. But as this is my first time adding code to DietPi, I'm sure there are things that I missed.

This has been tested both in a VM and on an RPi and appears to work ok. I assume there are features in the Web UI that will need tweaking since some things work differently when installed like this.

[dirkhh]

@MichaIng - I know we talked about this before, but I'm really struggling to get the build process to use my branch instead of your master branch... I believe it's supposed to be something like this:

export G_GITOWNER='dirkhh'
export G_GITBRANCH='adsb-feeder'
./dietpi-build -m 20 -v proxmox

But unless I hack around in the build script and flat out override the defaults, that doesn't appear to work for me.
Is there a trick I am missing?

[MichaIng]

It is two things:

• If you want the dietpi-installer and dietpi-imager scripts from your branch, then the exports are required.
• If you want the generated image to contain DietPi scripts from your branch, then it is:

  ./dietpi-build -m 20 -v proxmox -o dirkhh -b adsb-feeder

[dirkhh]

in the DietPi GitHub process - what does it mean that this PR is 'assigned' to me? Just making sure that I meet expectations 😎

[MichaIng]

Btw, we can also test things via GitHub Actions: https://github.com/MichaIng/DietPi/actions/runs/6030396640/job/16362120035

AFAIK you should also be able to trigger those on your fork, after enabling GitHub Actions there.

It currently fails since your repo does not contain the dietpi branch passed to git.

We can also add service or CLI tests for this software title to the test. But it has some limitations: Currently the Docker service cannot start within those containers, as they currently use the host network directly, while Docker requires a guest network (where it has "write" access to) to create its nested guest network, the routes/rules etc.

in the DietPi GitHub process - what does it mean that this PR is 'assigned' to me? Just making sure that I meet expectations 😎

I use this "assignment" as "this is from" for the PR. Usually there are multiple commits done on a PR, WIP, fixups etc, while I like to squash those into a single commit when merging into the dev branch. If there is no assignee and no explicit signoff in the commit message, it is then however shown as my commit (as I squashed it). Adding you as assignee assures that it is shown as your commit.

[dirkhh]

part of the functionality can be tested without docker. The adsb-setup python app runs without the container and we can check that the website does come up, etc.
which reminds me of a different question... by default this is designed to run on port 80. That seems like a questionable choice for this use case... any thoughts on that?

[MichaIng]

Indeed, as it is not served with one of the regular webservers, it would conflict with them when listening on port 80. So I'd prefer to have it on a different port and instead document how to change the port (in the Docker Compose YAML, I guess?), if no proxy is used anyway.

[dirkhh]

Do you have a registry of ports used? How do I find one that's free?

There is a bit of painful issue here that one of my 'outside' services won't work if this isn't on port 80... I need to redesign this... but someone who installs this locally really doesn't need that specific service as they obviously know the name/address of the DietPi machine (I have a neat little callback where once this starts up, you can go to my.adsb.im and get forwarded to your ADSB Feeder page - but that of course uses port 80.
And yes, this is all done in a completely privacy respecting manner, no log files kept, all data in a redis with 60s timeout.
But I think I'll simply disable this for the 'Software install' version of the ADSB Feeder and that will make the point moot.

[dirkhh]

Oh, and oddly... most of the Workflows don't show up in my fork... I only see the shellcheck one...

[dirkhh]

Btw, we can also test things via GitHub Actions: https://github.com/MichaIng/DietPi/actions/runs/6030396640/job/16362120035

AFAIK you should also be able to trigger those on your fork, after enabling GitHub Actions there.

Apparently not. All I get is shellcheck. None of the other actions.

It currently fails since your repo does not contain the dietpi branch passed to git.

Some days I am more on top of things than others. Today wasn't one of those days. I intentionally wanted to have a dedicated branch for the dietpi build to make sure that I can provide something that is tested and stable and all of that for the DietPi Software build. And then forgot to copy the branch that I had been working on over to that name 🤦🏻‍♂️🤦🏻‍♂️🤦🏻‍♂️

[dirkhh]

This is close but not quite there, yet. There are still a couple of differences between my own images and being "just an app" on a standard DietPi image that need to be addressed. And moving the config files added to the confusion.
But it's getting closer. I'll comment again when it passes my tests both on device and in a VM.

[dirkhh]

oh, one more question - I need a tempfs that I can share between a couple of containers (so that we reduce writes to the filesystem that may be on an SD card) - but that tempfs cannot have noexec set.
You set noexec on /run... is there a standard way to create another tempfs for this purpose, or could I remount /run without noexec, or how should I handle this?

[dirkhh]

OK, this is now reasonably well tested, both on a couple of different SBCs (OrangePi 5Plus and RPi4b) as well as in a VM.
The UI has all the "system" specific parts removed, but is otherwise identical to the existing ADSB Feeder Image builds which have a fair number of users, so while this is still somewhat 'niche', I think there are quite a few people who'd appreciate having it as an option on DietPi.

Please consider for merging.

[dirkhh]

one question... is there a way to register an app that should be called every time docker gets restarted?

[MichaIng]

With "app" you mean container? That should be the case OOTB, but can be toggled per-container if I am not wrong. If you mean another non-docker app, no natively that I am aware of. A ExecStartPost=/path/to/app could be added via docker.service drop-in config. Also it is possible to bind one systemd service to another to stat starting/stopping the first automatically starts/stops the second.

[dirkhh]

The adsb-feeder app will start between three and N containers, depending how many aggregators a user wants to feed or if they use multiple SDRs. Right now N is 14, but that may change over time.

I like the idea of chaining a service, that way this stays generic.

Thanks

[MichaIng]

We could also have a look into other Docker Conpose projects. I guess there is a common beat practice for starting/stopping all related containers gracefully with the Docker and/or containerd service. If I am not mistaken, containerd is actually the more important one for the containers to continue running, while Docker itself is more the communication/management API.

[dirkhh]

Since the actual setup app isn't in a container, all it needs is to find out when they go down and need to be restarted.
I'll look into what systemd already provides. Hopefully I'll avoid reinventing the wheel

[dirkhh]

Reading the man page is always a good start. Admittedly, this one had me confused at first, but with a bit of googling and some trial and error, I think I know what I need.
I'll add another service that is defined as BindsTo and After docker.service
And this service will then trigger the restart sequence for my containers and notify the setup Python app when docker disappears and subsequently reappears (for example when the user runs benchmarks, or with some other dietpi-xxx commands that temporarily stop docker.

I'd still appreciate any other review, test, etc, that is needed to get this PR merged as the main functionality is indeed working fine already - and I'd love to know if there are other issues / blockers from your perspective, @MichaIng

[dirkhh]

Thank you kindly. I will of course continue to monitor things in case there are any issues or suggestions - also always feel free to just @-tag me if you want my thoughts / help / input on stuff.

I am also planning to submit a few more changes for the workflow challenges that I feel there are today :)

[andreagdipaolo]

I don't know if that's the right place to write about this, but after installing this software all my running containers have been stopped and volumes deleted. Maybe it could be handled differently. Fortunately I didn't lose any critical data, and I am not even 100% sure that this is the cause, but It's very likely that the script somehow interferes with other containers.

Do you have any clue? @dirkhh

[MichaIng]

I think I might know the reason: When installing Docker via dietpi-software, the data directory in /etc/docker/daemon.json is set to /mnt/dietpi_userdata/docker-data. By default it is /var/lib/docker, so if you installed Docker just via official installer or APT previously, all containers and data remain unused in /var/lib/docker.

Quite interesting that this issue never came up before, but I think we should handle it gracefully. Either we move /var/lib/docker to /mnt/dietpi_userdata/docker-data if it exists, or we keep using the default dir, if actual data/containers/volumes are contained already, or we ask the admin what to prefer (still requiring a default choice for non-interactive installs). Any thoughts?

So @andreagdipaolo please verify that all your previous containers are in /var/lib/docker, and if so, and you did not do any significant setup with ADS-B Feeder, try this:

sudo mv /mnt/dietpi_userdata/docker-data /mnt/dietpi_userdata/docker-data.bak
sudo mv /var/lib/docker /mnt/dietpi_userdata/docker-data
sudo dietpi-software reinstall 141

And if all works well, no container or something is missing:

rm -R /mnt/dietpi_userdata/docker-data.bak

[andreagdipaolo]

Sadly they are not there. Indeed the files are there, but they are empty because volumes have already been re-created when I ran a docker compose up in order to update and restart the containers. Also, I usually do a prune after every successful upgrade.

After launching your three commands the aiplane containers have not gone up, and my home assistant instance is broken. Probably moving the paths broke something in my database configuration. Immich is refusing the connections and Z2M is dead.

My Docker installation has been via dietpi-software.

[MichaIng]

Hmm, nothing else should directly access to the Docker data directory, but only Docker itself. So I cannot imagine how it could have broke something. But of course you can just move things back and also change the data directory in /etc/docker/daemon.json back.

My Docker installation has been via dietpi-software.

Ah, so, there was no content in /var/lib/docker then? Okay then I don't know either. Probably the adsb-nonimage.service is doing a too intrusive housekeeping?

my home assistant instance is broke

You use HA supervised?

Just that we do not forget: #6660

[andreagdipaolo]

So I cannot imagine how it could have broke something.

I don't know. Containers are up (except those managed by adsb.im) but nothing seems to run correctly.

Ah, so, there was no content in /var/lib/docker then?

Files were all there, but the one I was looking for (the immich database) was empty, probably because after a down -v and a prune it got nuked and recreated with my following update.

You use HA supervised?

I use their docker container. My Docker ps looks like this:

[andreagdipaolo]

I believe my data was in the dietpi_userdata folder, but something was in the var/lib folder too.
Probably if I restore the backup my containers will work again. But why was the structure present in the var/lib directory too?

[dirkhh]

It would be good to understand in more detail what for disconnected and why.
Was docker installed via dietpi-software or separately?
I think I know one spot where the current ads.im code might do rude things to existing volumes.
I'll address this shortly (I'm on a plane right now).

[MichaIng]

But why was the structure present in the var/lib directory too?

Not sure. I guess it is also possible to define a data/image dir explicitly for a specific container. Probably you can find out whether it really is still in use and which container it is used by, checking timestamps.

Leaving a symlink ln -s /mnt/dietpi_userdata/docker-data /var/lib/docker in place on dietpi-software Docker installs, just in case anything is still accessing/expecting the old dir, is also a good idea I think.

[andreagdipaolo]

Now everything started working without doing anything.
adsb.im containers appeared in the docker ps and home assistant and Z2m work fine.
Immich still refuses connections but maybe I will investigate on that later.

I'm quite confused by this behaviour.

Was docker installed via dietpi-software or separately?

If I check in dietpi-software both docker and docker compose have the * , so I believe they were installed in that way.
Can I check elsewhere?

Also, adsb.im is stuck at version 16.3 even after reinstalling. Do dietpi software releases take more time to follow github releases?

[dirkhh]

The ADS-B feeder pulls docker and docker compose.

My question was how you initially installed docker, before installing the ADS-B feeder app

[MichaIng]

Also, adsb.im is stuck at version 16.3 even after reinstalling. Do dietpi software releases take more time to follow github releases?

The host-side services, scripts and config files are updated on reinstall, but the container image itself is not, as far as I can see. Is systemctl start adsb-feeder-update probably something to add to our documentation @dirkhh?

[dirkhh]

The dietpi release of 0.17.x will happen tonight

[andreagdipaolo]

My question was how you initially installed docker, before installing the ADS-B feeder app

As I said, I don't remember, but if I see them with the * in dietpi software I suppose they were installed in that way.
Can I check elsewhere?

[dirkhh]

Also, adsb.im is stuck at version 16.3 even after reinstalling. Do dietpi software releases take more time to follow github releases?

The host-side services, scripts and config files are updated on reinstall, but the container image itself is not, as far as I can see. Is systemctl start adsb-feeder-update probably something to add to our documentation @dirkhh?

You can update the feeder from the web UI.
I need to make sure that works without a problem before I enable the newest version for users on DietPi.

[MichaIng]

if I see them with the * in dietpi software I suppose they were installed in that way.

This would be true as well if you did install Docker differently before. One possible indicator:

ls -l /etc/apt/keyrings/docker.gpg

That /var/lib/docker exists and is not empty is probably prove already that it was installed differently before.

[andreagdipaolo]

ls -l /etc/apt/keyrings/docker.gpg

-rw-r--r-- 1 root root 2760 Jul 27 21:14 /etc/apt/keyrings/docker.gpg

Does it mean anything?
Anyway I can tell for sure that sometimes I update it via the sudo apt upgrade. I don't remember if Dietpi-software can appear there or it must be reinstalled via the tool.

[MichaIng]

Yes, this means you installed Docker via their installer in the first place, indeed.

[andreagdipaolo]

Yes, this means you installed Docker via their installer in the first place, indeed.

Oh, ok. So dietpi-software detects it even if I didn't install it via that?

[MichaIng]

Since Docker is defined as dependency for ADS-B Feeder in dietpi-software, it installs it on top of it. It is basically a no-op, since it does so with the same APT repository and package, but we install the APT key to /etc/apt/trusted.gpg.d/dietpi-docker.gpg instead of /etc/apt/keyrings/docker.gpg and do some config changes, e.g. the data-root change which caused the issue here.