Monitoring system performance, activity & email alerts

What ever your setup, once running, you should monitor your resource usage, as running out of CPU will cause trunk recorder to drop voice signals and create garble/noises. You can see this happening in the console with printing of OOOOOO's FAQ#i-am-seeing-a-lot-of-0000s-or-oooos-in-my-console And you will need to either upgrade your system or reduce allocated bandwidth/recorders.

The simplest way is just with "top" command, where you can see your load average over 1,5 & 15 minute average as well as momentary cpu usage. CPU usage is 100% per cpu core and load is 1.0 per core, so 4 core will maxed out at 4.0 load and 400% usage, but you'll start to see OOOO's before that.

Another simple terminal program that can show this (load) over time in a graph is 'ttyload' but it will only show you history while you're running it.

For longer term monitoring you can use 'munin' which stores records/graphs over day/week/month for all system resources. (keep in mind with this though that it currently logs at 5 minute interval so it wont necessarily catch intermittent high load generated during a call recording if your system is not very active, this is where watching live in terminal while a recording is taking place is still necessary.

You should also and likely will automatically check the output recordings as there may be weak signal or other problems, they can be optionally deleted or not.

Here are some rough notes from getting munin working under debian,

install munin and apache2 packages
ln -s /etc/munin/apache24.conf /etc/apache2/conf-enabled/munin.conf
restart apache
open http://device_ip/munin in browser

Munin comes default with a bunch of plugins installed & enabled, so you should get graphs generated within 5-10mins, plugins/graphs can be toggled with symlinks in /etc/munin/plugins/ followed be a service restart.

Munin Trunk Recorder call activity plugin

It works simply by using uploadScript setting to touch an empty file in a chosen folder for each call, then each 5 mins when munin calls this plugin it counts the files in folder to get count of calls in that time, and then deletes all the files. This number is reported in graph.

Create the following file and symlink it

#!/usr/bin/env python3
#/usr/share/munin/plugins/trunkrecorder-munin.py
import os
import sys
import datetime

timeSpan = 5 #minutes
logDir = '/var/log/munin/trunkrecorder-munin'


if('config' in sys.argv):
	print('graph_title Trunk Recorder Calls')
	print('graph_vlabel Call Count')
	print('graph_printf %.0lf' )	
	print('graph_category Trunk')
	print('trunk.label All Calls')
	print('trunk.draw AREA')
	print('trunk.critical 1:') #alert if falls to 0

	sys.exit()


count = 0
for filename in os.listdir(logDir):
	count+=1
	os.remove(logDir+'/'+filename)

print("trunk.value " + str(count))

ln -s /usr/share/munin/plugins/trunkrecorder-munin.py /etc/munin/plugins/

then add the following to bottom of /etc/munin/plugin-conf.d/munin-node

[trunkrecorder-munin.py]
user root
group root

restart munin

/etc/init.d/munin-node restart

to see if plugin enabled run:

munin-node-configure

to test run plugin to see if working:

munin-run trunkrecorder-munin.py

Here is the python code that must be called by uploadScript feature of trunk recorder, either add it to your existing python script or add one

##############################
#log for munin

logDir = '/var/log/munin/trunkrecorder-munin'
#must exist and be outside /home for munin unless cautiously change ProtectHome setting in systemd service file
#also munin user must have write access to it, or adding a setting in /etc/munin/plugin-conf.d/munin-node to match.

#touch file for later counting    
os.close(os.open(logDir+'/'+os.path.splitext(os.path.basename(sys.argv[1]))[0], os.O_CREAT))

#log for munin
###############################

You must also add the following mount to your trunk recorder docker container if using docker so the created files end up on the host and not just in the container:

- '/var/log/munin:/var/log/munin'

Munin can optionally have alarms set that can trigger for example an email. I use swaks package to send the mail via a smtp account i have. Edit /etc/munin/munin.conf :

contact.email.command swaks --to [email protected] --from [email protected] --header 'Subject: Munin-notification for ${var:plugin}' --body 'Current value of ${var:label} is ${var:value}' --server smtp.server.com.au:587 --auth LOGIN --auth-user "[email protected]" --auth-password "password" -tls

the alerts levels by default are set in the plugin file eg

print('trunk.critical 1:') #alert if falls to 0 to warn me that something likely broke

I found the following command to send a test mail on demand:

su - munin --shell=/bin/bash -c "/usr/share/munin/munin-limits --contact email --force"

more details http://guide.munin-monitoring.org/en/latest/tutorial/alert.html