Homer 7 Stack Installation on Debian 9 Stretch

HOMER 7

Major Changes

Capture Server was changed from Kamailio to Heplify-Server.
PostgreSQL was used instead of MySQL due to better partitioning, native JSON support.
Support Natively different Metrics Stacks. (TICK, Prometheus, Loki, ElasticSearch...)
Better APIs

Overview of Components

There are quite a few components which make up a complete Homer 7 stack. Below is the list for this guide. These are the links to each individual component for easy reference.

PostgreSQL 10

PostgreSQL is where the HEPlify-Server stores all of the raw data that it ingests from the HEPlify Capture Agent. My suggestion is to size this appropriately to your environment. A smaller environment will probably not need as much resources as I've specified here, whereas a larger environment will probably require more. If you can back the database with fast disk that is helpful as well.

VM Specs
Minimum 4 CPUs
Minimum 16GB RAM
Minimum 1TB Storage Space

Package Requirements

apt-get install sudo

Note: Make sure your PostgreSQL Data directory is configured to point to a very large data storage space. By default PostgreSQL will put all data into the same directory as the configuration files.

Install and Start PostgreSQL-10

Add the key.
wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O- | sudo apt-key add -
Add repository.
echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" > /etc/apt/sources.list.d/postgresql.list
Install postgresql-10 from the repo.
apt-get update && apt-get install -y postgresql-10
Enable the services.
systemctl enable postgresql
systemctl start postgresql
Create the database and set appropriate permissions.
sudo su - postgres
createuser homer --pwprompt
createdb -O homer homer_config
createdb -O homer homer_data
psql
postgres=# GRANT ALL PRIVILEGES ON DATABASE "homer_config" to homer;
postgres=# GRANT ALL PRIVILEGES ON DATABASE "homer_data" to homer;
Modify the connection file to allow inbound connections to the postgresql services(If postgresql is on different server than homer-app and heplify-server).
vi /etc/postgresql/10/main/pg_hba.conf
Add to the bottom of the file this line:
host all all [IP of HEPlify-Server]/32 password
host all all [IP of homer-app server]/32 password
Modify the postgresql.conf configuration file and set the following variables to have a good running server.

I found this PostgreSQL tuning guide: http://linuxfinances.info/info/quickstart.html
File Path: /etc/postgresql/10/main/postgresql.conf
listen_addresses = '*' under the -Connection Settings- header.
shared_buffers = 1024MB
effective_cache_size = 12GB
Note: This will likely be most of the servers physical memory, if postgres is installed by itself.
max_locks_per_transaction = 1000
data_directory = '[path to data directory]'

Restart the postgresql-10 service to commit the changes.
systemctl restart postgresql

Homer Web App

This is the Web application which displays the Homer WUI to the user.

System Requirements:

2 vCPUs
8GB RAM
40GB Storage

Package Requirements

apt-get install curl git sudo software-properties-common make g++

Clone the GitHub repository to /opt/homer-app
cd /opt
git clone https://github.com/sipcapture/homer-app
Install NVM from GitHub (Source: https://github.com/nvm-sh/nvm#installation-and-update)
curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs
Install npm dependencies.
cd /opt/homer-app
npm install --unsafe-perm && npm install --unsafe-perm -g knex eslint eslint-plugin-html eslint-plugin-json eslint-config-google
Configure the Postgres Database connection in the config.js file.
vi /opt/homer-app/config.js
Set host: value to the IP address of the PostgreSQL server.
Set user: and password: values to a user with access to the homer_data and homer_config databases.

This is an example of what the configuration block will look like for the PostgreSQL configuration.

  	const pgsql = {
  	  host: '[postgre_server_ip]',
  	  user: '[postgre_user]',
  	  port: 5432,
  	  password: '[postgre_password]',
  	  charset: 'utf8',
  	  timezone: 'utc',
  	  pool: {
  	    afterCreate: function(connection, callback) {
  	      connection.query('SET timezone = "UTC";', function(err) {
  	        callback(err, connection);
  	      });
  	    },
  	  },
  	};

Create homer_data and homer_config databases and insert default data by using below commands.
knex migrate:latest
knex seed:run
Build the app bundle using webpack. Do this within the /opt/homer-app directory.
npm run build
Setting up the homer-app as a service.

Add the following service file in /etc/systemd/system/

Name: homer-app.service
Put this in the file:

  	[Unit]
  	Description=Homer 7 UI
  	After=network.target
  	
  	[Service]
  	WorkingDirectory=/opt/homer-app
  	ExecStart=/usr/bin/node --max_old_space_size=2048 /opt/homer-app/bootstrap.js
  	Restart=on-failure
  	RestartSec=10s
  	Type=simple
  	
  	[Install]
  	WantedBy=multi-user.target

Reload the systemctl services and start the service.
systemctl daemon-reload
systemctl enable homer-app
systemctl start homer-app

You can now browse to the server IP in order to verify that it's working properly.

HEPlify-Server

The HEPlify-Server is the "traffic cop" of all of the data that comes into the Homer 7 stack. It is this component which ingests all the data, then sends it out to all the other components in the proper format.

System Requirements:

8 vCPUs
16GB RAM
100GB Storage

Package Requirements

libpcap-dev
Go programming language

Install Golang.
cd /usr/src/
curl -O https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz
tar xvf go1.12.7.linux-amd64.tar.gz
sudo chown -R root:root ./go
sudo mv go /usr/local

First, set Go's root value, which tells Go where to look for its files: vim ~/.profile
export GOPATH=$HOME/work
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
Next, refresh your profile by running:
source ~/.profile

Install the Heplify-Server Binaries

cd to opt
cd /opt
Clone the github repo into the /opt directory.
git clone https://github.com/sipcapture/heplify-server
Build the services using Go.
go build cmd/heplify-server/heplify-server.go
Modify the configuration file to point to the proper Postgres services.
cp /opt/heplify-server/example/homer7_config/heplify-server.toml /opt/heplify-server/
vi /opt/heplify-server/heplify-server.toml

Set DBAddr to "[ip of postgres]:5432"
Set DBUser and DBPass according to what you configured when setting up the PostgreSQL-10 database user.
Set DBWorker equal to the number of CPUs in the system.
Set ESDiscovery to false if you do not plan on using Elasticsearch.
Set LogDbg to "hep,sql"
Set LogLvl to "warning"
Set DiscardMethod to ["OPTIONS","NOTIFY"]
Set PromAddr to "0.0.0.0:9096"

Note: A good guide on all of the HEPlify-Server configuration parameters can be found in the Wiki here: HEPLIFY-SERVER-Settings

Setup the system service to run the heplify-server binary when the system loads.
cp /opt/heplify-server/example/heplify-server.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable heplify-server
systemctl start heplify-server

You can verify the service is running with systemctl status heplify-server
You can verify that the service is able to connect to the postgre server by looking at the /opt/heplify-server/heplify-server.log file.

If everything is working properly the Heplify-Server service is now ready to ingest data from the Heplify capture agent service. You can verify this by using the tail command to look at the heplify-server.log file. If working properly you will see the packets flowing through this file.

tail -f /opt/heplify-server/heplify-server.log

You can verify that PostgreSQL is writing data into the database by using the top command. There should be postmaster commands running in the output.

TICK Stack Installation

InfluxDB is an Open Source Time Series Database Platform for storing Time Series Data, these are metrics & Events collected from different devices. It is a product of InfluxData and part of TICK Stack – which comprises:

T – Telegraf; This is an agent written in Go for collecting performance metrics from the system it’s running on and the services running on that system. The collected metrics are output to InfluxDB or other supported data stores.
I – InfluxDB
C – Chronograf; A web application written in Go and React.js that provides the tools to visualize your monitoring data and easily create alerting and automation rules.
K – Kapacitor A data processing engine. It can process both stream and batch data from InfluxDB.

System Requirements:

4 vCPUs
8GB RAM
1TB Storage

Package Requirements

Add repository for the TICK Stack
echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
Install the TICK Stack.
sudo apt-get update && sudo apt-get install influxdb kapacitor telegraf chronograf
Start and Enable the Services.
sudo systemctl start influxdb && sudo systemctl enable influxdb
sudo systemctl start telegraf && sudo systemctl enable telegraf
sudo systemctl start kapacitor && sudo systemctl enable kapacitor
sudo systemctl start chronograf && sudo systemctl enable chronograf

The HEPlify Capture Agent

HEPlify Capture needs to be installed on the system you would like to capture the traffic on. The HEPlify Capture captures the data, sends it to the HEPlify-Server to be ingested which sent it out to the other components of the stack.

OS Packages:

The Go programming Language.
PCAP Libraries

Installation

Install your prerequisites.
apt install git make libpcap-dev curl sudo
Clone the github repo
cd /opt/ && git clone https://github.com/sipcapture/heplify
Run the make file in the cloned location with the make command.
cd heplify && make
Note: Heplify capture requires root permissions to run.

Testing
You should now be able to start the heplify capture by running the heplify executable file. The output should be sent to the screen, and the heplify.log file should show the most recent information.

Service Installation

Copy the example service file to the proper spot in the file system.
cp /opt/heplify/example/heplify.service /etc/systemd/system/
Modify the executable path in the file to match what you want to be capturing. This is where you would modify it to specify which physical interface to listen on, as well as what server to send the captured packets to.

This is what the production hep capture service file looks like. Note that [interface_name] is the system name of the interface which will be listening and will be the monitor destination below.

  [Unit]
  Description=Captures packets from wire and sends them to Homer
  After=network.target
  
  [Service]
  WorkingDirectory=/opt/heplify
  ExecStart=/opt/heplify/heplify -i [interface_name] -hs [ip_of_heplify_server]:9060 -m SIPRTCP
  ExecStop=/bin/kill ${MAINPID}
  Restart=on-failure
  RestartSec=10s
  Type=simple
  
  [Install]
  WantedBy=multi-user.target

Enable the service.
systemctl daemon-reload
systemctl enable heplify
systemctl start heplify
Validate that the service is running by using systemctl status heplify
The log is written to /opt/heplify/heplify.log

Useful Information

Here is some other useful information that I have collected for everyone's reference.

KPIs

Key Performance Indicators which can be looked up here: https://tools.ietf.org/html/rfc6076

Session Establishment Ratio (SER)

This metric is used to detect the ability of a terminating UA or downstream proxy to successfully establish sessions per new session INVITE requests.

Session Establishment Effectiveness Ratio (SEER)

This metric is complimentary to SER, but is intended to exclude the potential effects of an individual user of the target UA from the metric.

Session Completion Ratio (SCR)

A session completion is defined as a SIP dialog, which completes without failing due to a lack of response from an intended proxy or UA.

Ineffective Registration Attempts (IRAs)

Ineffective registration attempts are utilized to detect failures or impairments causing the inability of a registrar to receive a UA REGISTER request.

Ineffective Session Attempts (ISAs)

Ineffective session attempts occur when a proxy or agent internally releases a setup request with a failed or overloaded condition.

The Homer Dashboard

The Homer dashboard allows you to grab all the signaling details for a set of calls. Searching is a little non-intuitive. Here are some suggestions to make use of the search features.

Set the Time appropriately on the top-left of the screen where it says "Today". My suggestion is to use a specific time frame.
Next enter either the To number or the From number in the Search query and click on Search. This will bring up a new window with all of the results from the query.
To searches work best.
From the resulting values, you can click on the SID to bring up the details from the call.
Flow = SIP Ladder
QoS = RTCP QoS Values
Export allows you to export the information into a PCAP file.