Feedback on the new Configuration Tool approach

[JRG1956]

Hi Jasper,

As requested, this is to provide some further feedback on the new configuration tool and config files approach.

Clearly this is still early-stage, but I think it will help a lot of users who are new to AIS-catcher as it evolves.

For context, I'm new to ship tracking and using RTL-SDR dongles but have been feeding a number of aircraft tracking websites from a Raspberry Pi4B using an Airspy Mini receiver for four years. I use wiedehopf's readsb, tar1090 and graphs1090 software - his approach to configuring the apps probably influences my opinions. I initially set up my AIS-catcher feed using abcd567a's installer (https://github.com/abcd567a) with an RTL-SDR dongle on a Pi4B - it is working well.

I'm experimenting with AIS-catcher and the config tool on a Pi5B with an Airspy R2 attached to an ads-b antenna without an LNA, which much to my surprise, is actually receiving some AIS messages. My thoughts so far:

If I've understood correctly, a lot of parameters appear to be able to be placed in either the config.json or config.cmd file, with the config tool populating the config.json - At this early stage, the documentation doesn't make it clear if there is a preferred location for the parameters, and if one file has priority where the parameter has been specified in both files. For someone like me coming from a flight tracking background, the inclination is to edit the config.cmd file because the json structure is unfamiliar.

I managed to get the test rig running by using the config tool to specify the receiver, etc and then adding entries to config.cmd for the other parameters. The contents of the two files were

{
"config": "aiscatcher",
  "version": 1,
  "sharing": "false",
  "screen": 0,
  "verbose": true,
  "sharing_key": "",
  "input": "AIRSPY",
  "serial": "735863DC3086xxxx"
}

(serial number and location edited for privacy) and

### Test config.cmd file

-gm LINEARITY 21 BIASTEE off
-s 6000000

-m 2
-v 60
-M DT
-o 0

-u 127.0.0.1 10110
-N 8383 FILE stat.bin BACKUP 10
-N PLUGIN_DIR /usr/share/aiscatcher/my-plugins
-N STATION Breamlea LAT -38.2xx LON 144.3xx SHARE_LOC off

(If the parameters look a bit strange, it is because I carried them over from my RTL-SDR setup, and I also sometimes like to make default settings explicit.) These settings work fine:

● ais-catcher.service - AIS-catcher Service
     Loaded: loaded (/etc/systemd/system/ais-catcher.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-07-23 17:07:27 AEST; 1min 32s ago
   Main PID: 763 (AIS-catcher)
      Tasks: 6 (limit: 4450)
        CPU: 16.241s
     CGroup: /system.slice/ais-catcher.service
             └─763 /usr/bin/AIS-catcher -C /etc/AIS-catcher/config.json -gm LINEARITY 21 BIASTEE off -s 6000000 -m 2 -v 60 -M DT -o 0 -u 127.0.0.1 10110 -N 8383 FILE stat.bin BACKUP 10 -N PLUGIN_DIR /usr/share/aiscatcher/my-plugins -N STATION....

Jul 23 17:07:28 pi51 bash[763]: Warning: sample rate 6000K upsampled to 6144K.
Jul 23 17:07:28 pi51 bash[763]: UDP: open socket for host: 127.0.0.1, port: 10110, filter: OFF, JSON: OFF
Jul 23 17:07:28 pi51 bash[763]: Server: reading statistics from stat.bin
Jul 23 17:07:28 pi51 bash[763]: TCP Server: start thread at port 8383
Jul 23 17:07:28 pi51 bash[763]: HTML Server running at port 8383
Jul 23 17:07:28 pi51 bash[763]: Server: starting backup service every 10 minutes.
Jul 23 17:07:28 pi51 bash[763]: Device    : AIRSPY
Jul 23 17:07:28 pi51 bash[763]: Settings  : rate 6000K format CF32 linearity 21 biastee OFF
Jul 23 17:07:28 pi51 bash[763]: Model #0 -> (Src: 1, Grp: 1): [AIS engine v0.59] ps_ema ON afc_wide ON droop ON fp_ds OFF dsk OFF
Jul 23 17:08:28 pi51 bash[763]: [AIS engine v0.59 #0-0]                 received: 2 msgs, total: 2 msgs, rate: 0.0332869 msg/s

However if I also specify the serial number at the start of the config.cmd:

-d 735863DC3086xxxx

ais-catcher fails:

× ais-catcher.service - AIS-catcher Service
     Loaded: loaded (/etc/systemd/system/ais-catcher.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-07-23 17:04:24 AEST; 1min 31s ago
   Duration: 26ms
    Process: 908 ExecStart=/bin/bash -c /usr/bin/AIS-catcher -C /etc/AIS-catcher/config.json $(/bin/grep -v "^#" /etc/AIS-catcher/config.cmd | /usr/bin/tr "
" " ") (code=exited, status=255/EXCEPTION)
   Main PID: 908 (code=exited, status=255/EXCEPTION)
        CPU: 24ms

Jul 23 17:04:24 pi51 systemd[1]: ais-catcher.service: Scheduled restart job, restart counter is at 5.
Jul 23 17:04:24 pi51 systemd[1]: Stopped ais-catcher.service - AIS-catcher Service.
Jul 23 17:04:24 pi51 systemd[1]: ais-catcher.service: Start request repeated too quickly.
Jul 23 17:04:24 pi51 systemd[1]: ais-catcher.service: Failed with result 'exit-code'.
Jul 23 17:04:24 pi51 systemd[1]: Failed to start ais-catcher.service - AIS-catcher Service.

I also tried specifying a different serial number from the json file in the cmd file, and that also failed.

I haven't tried doing this with other parameters.

So the current approach appears to work as long as users only specify things in one place.

As you say in the readme, this is at an early stage - hope the feedback is useful.

John

[jvde-github]

Thanks, I was triggered because users find the command line much more intuitive than the JSON, but the JSON is easier to set with a menu system. I agree, this needs to be streamlined/clarified. Will have a look, thanks! More feedback always welcome :-)

[jvde-github]

It looks like if you specify the dongle in the json and cmd-line, it will set up two parallel receivers, but since the dongle can be opened only once it terminates with an error:

AIS-catcher -C /etc/AIS-catcher/config.json  -d 00000001
AIS-catcher (build Jul  8 2024) v0.59-210-ga9a68a08
(C) Copyright 2021-2024 jvde-github and other contributors
This is free software; see the source for copying conditions.There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Searching for device with SN 00000001.
Found Rafael Micro R828D tuner
RTL-SDR Blog V4 Detected
Searching for device with SN 00000001.
usb_claim_interface error -6
Error: RTLSDR: cannot open device.

So in principle it will read from command line and applies settings in order they appear there and then followed by the parameters set in the config file. So some parameters will be overwritten by the config, e.g. (screen or '-o' will be the latest) but other parameters are additive, you can run like -d 001 -d 002 with two dongles, but also define multiple webviewers (-N 8101 -N 8102) Having it both in the parameters and config file will open two receivers.

That is how it is now and it is confusing. Maybe for the menu system I should a) keep it simple and only manage start/stop of the program and ask the user to set the other parameters in the cmd-file or b) allow for both and accept we get some confusion like just described.

[jvde-github]

also think we need to add an option to disable a systemd service (next to start and stop)...

[JRG1956]

Thanks for the explanation - I was wondering why when I set -o 3, I didn't see any individual messages in the log, just the 60 second summary lines from - v 60. The json was clobbering the cmd-file.

I think that as an interim step, limiting the menu system to start/stop/disable might be the best approach while you think about whether that approach is best for other options. (If you do enable specifying other parameters in the menu, maybe ignore those parameters if they are also set in the cmd-file?)

I suspect most users will be comfortable with specifying everything in the cmd-file. I suggest telling users to specify everything in the cmd-file or the json but not to populate both - in case some users want to manually edit the json.

I also think a simple explanation of the cmd-file format (lines starting with a # or space are ignored) and providing a short example of the contents of a file would help users understand what is needed.

[jvde-github]

Yes, I think I will follow your suggestion, mostly because it keeps things also easy to implement. That means the menu is only used for starting and stopping the daemon. Also, perhaps best to use "source build" as default.

[JRG1956]

Yes I was wondering which installation build I should use - I ended up choosing the "source build". Do you need to also use the same build for all subsequent updates?

An extra suggestion re the cmd-file is to create it with a basic default set of options for an RTL-SDR dongle (I assume the majority of users have these?) with comment lines explaining the options - much like this default file from graphs1090:

https://github.com/wiedehopf/graphs1090/blob/master/default

It might be easier to comment out parameters and require the user to edit and uncomment them if they need to. Something along the lines of the following would work with your current cmd-file structure:

# To share data with AIS-catcher uncomment the following line and replace <key> with your sharing key
#-X <key>```

[jvde-github]

Ok, removed most script and kept it one that installs from source with an example config.cmd file._

[JRG1956]

Thanks - I've just installed the latest version on a raspi4 with a fresh version of bookworm to emulate a new user installing from scratch. I got the following result:

× ais-catcher.service - AIS-catcher Service
     Loaded: loaded (/etc/systemd/system/ais-catcher.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-07-29 16:46:56 AEST; 1min 11s ago
   Duration: 44ms
    Process: 2897 ExecStart=/bin/bash -c /usr/bin/AIS-catcher -C /etc/AIS-catcher/config.json $(/bin/grep -v "^#" /etc/AIS-catcher/config.cmd | /usr/bin/tr "
" " ") (code=exited, status=255/EXCEPTION)
   Main PID: 2897 (code=exited, status=255/EXCEPTION)
        CPU: 46ms

Jul 29 16:46:56 pi45 systemd[1]: ais-catcher.service: Scheduled restart job, restart counter is at 5.
Jul 29 16:46:56 pi45 systemd[1]: Stopped ais-catcher.service - AIS-catcher Service.
Jul 29 16:46:56 pi45 systemd[1]: ais-catcher.service: Start request repeated too quickly.
Jul 29 16:46:56 pi45 systemd[1]: ais-catcher.service: Failed with result 'exit-code'.
Jul 29 16:46:56 pi45 systemd[1]: Failed to start ais-catcher.service - AIS-catcher Service.
jrg@pi45:~ $ cat /etc/AIS-catcher/config.json
{ "config": "aiscatcher", "version": 1 }
jrg@pi45:~ $ cat /etc/AIS-catcher/config.cmd
# AIS-catcher configuration
# Uncomment and modify options as needed

# Select device for decoding
-d:0

# Share feed with aiscatcher.org
-X

# Open webviewer at port 8100
-N 8100

# Uncomment to set location and allow viewing in webviewer
# LAT 12.3 LON 4.3 SHARE_LOC ON

# Uncomment to share data via UDP
# -u 127.0.0.1 10110

# Uncomment to set up a TCP listening server
# -S 5012

I didn't have a receiver attached, but should the ais-catcher service be disabled on new installs?

I haven't looked at the install script, but assume that if it finds a config.cmd file, it leaves it alone and respects the current enabled/disabled setting for the ais-catcher service?

[JRG1956]

Answering my own questions re the install/update script:

• as expected, config.cmd is preserved.
• if ais-catcher was disabled, it is re-enabled and started.

I suppose that makes sense, as you would expect most people running an update would want the application to run at the end of the process :-)

A question about the -X (blank) line in config.cmd - this seems to result in a connection to aiscatcher.org:
Jul 29 18:05:04 pi45 bash[762]: TCP (aiscatcher.org:4242): connected to server.
Is this intended?

Overall, I think the cmd-file approach works well and should be easy for most users to understand.

[jvde-github]

I thought that I answered this but don't see the reply. The aiscatcher.org:4242 connection is the community feed driven by -X indeed.

I have fixed the script so it does not auto-enable and instead mention how to enable in the documentation. This potentially avoids dongle conflicts when the user first installs. It retains though the enable and start state of the service.

[JRG1956]

Thanks for the updated script and approach - it works well.

I've just tested it by moving my feeder from a Pi4 to a fresh build on a Pi5 and have that up and running without any issues. Just to give you a benchmark for installing on a Pi5 - the whole first installation process took 2 minutes and 53 seconds. The Pi5 was already running flight feeder software with an overall load of 30% when I installed AIS-catcher! It is happy running a RTL-SDR dongle for AIS-catcher and an Airspy R2 for readsb/tar1090 at the same time.

[Taxom]

I'm one of those who don't like configuration using JSON. I find it very overloaded and hard to read. I prefer configuring using regular text files. "parameter" "value" "#comment"
like this:

[jvde-github]

Thanks! I have heard this feedback before. I have updated the script. If you install with the script, you can copy paste this into /etc/AIS-catcher/config.cmd and should be ok.

[Taxom]

Hi, I'm currently using the script from https://github.com/abcd567a/install-aiscatcher for installation (updates), and I don't want to break anything by changing the installation script.

[jvde-github]

If you are up and running there is no point to switch indeed.

[g7ruh]

Hi Jasper,

I too dislike JSON as per @Taxom above. I much prefer parameter # comment and to allow empty lines between items, this allows grouping of entries: e.g. basic config and then feeder sites

Regards
Roger

[dreua]

Have you considered using yaml instead of json? It is much easier to read and edit by humans but holds the same data structures as json. Depending on your environment (language/libraries) it could be very easy to switch to yaml or even support both json and yaml for the best compatibility. It also allows for comments as requested in another thread here.

[jvde-github]

Haven't looked into it (or at least did not think about it, so thans). It is a good alternative if I implement some options to define settings via a menu system. Will look into it at that time.