-
Notifications
You must be signed in to change notification settings - Fork 21
Jesse's "checknode" Monitoring Script
TOMO tip address: 0x19274dD87c3f824f004184E59AcB148b11D0B244
Version 1.2: March 8th 2019
Update v1.1 - Mar 1st 2019: added error handling in case TOMO Master is down
Update v1.2 - Mar 8th 2019: updated path to ssmtp to include full /usr/sbin path
Email or SMS alerts about the status of the node. If the node's status changes from "masternode" to proposed or slashed or if the current block hasn't updated in over 60 seconds, this script will email/SMS you an alert.
*Required binary packages: jq, ssmtp
Note: This guide assumes you do not have 2FA enabled on the Gmail account that you will use. You may want to create a new Gmail account just for this script, as you will be entering your password in plaintext into a configuration file on your server.
Make sure you are logged in with your masternode user; the one you installed "tmn" with.
First we need to install and setup SSMTP, which allows you to use your Gmail account to send email from the Linux command line. This also eliminates the need to have a full MTA (Message Transfer Agent) installed on the server. If you already have an MTA installed, you will want to use that instead of SSMTP.
By default, Ubuntu does not have an MTA installed, so if you never set one up, proceed with the SSMTP installation here:
Note: If you never set the server's hostname, SSMTP will fail to install, so make sure you've set a hostname for your server. You can use the following guide:
https://linuxize.com/post/how-to-change-hostname-on-ubuntu-18-04/
- Install "ssmtp" and "jq", a JSON parser needed for the "checknode" script:
sudo apt install ssmtp jq- Edit the ssmtp.conf to include your gmail account details:
sudo nano /etc/ssmtp/ssmtp.conf- Make sure only the following lines are present in the file, change "user" and "password" to match your gmail account:
[email protected]
mailhub=smtp.gmail.com:587
[email protected]
AuthPass=password
UseSTARTTLS=YES- Test that you can send mail from the command line (change "user" to match your address):
echo "test" | ssmtp [email protected]You should receive an email with no subject and "test" as the body. If, instead, you get an error like below, you will need to configure some things in your gmail settings to allow scripts to authenticate with your gmail account:
ssmtp: Authorization failed (535 5.7.8 https://support.google.com/mail/?p=BadCredentials p47sm4360595eda.31 - gsmtp)You can use the link provided in the error and follow the instructions. Each time you make a change in your gmail settings, try sending a message using the above command to see if the change you made worked. I had to do a few things in my Gmail settings before this worked. I can't pinpoint the exact step that worked, as I didn't test after each change.
Once you have SSMTP working correctly, we can get started on the "checknode" script and cronjob setup.
The very first thing you need to do is figure out the correct email address to use in order to text your phone number. Most, if not all, cellphone carriers have this ability. Here is a list of some common US carriers:
https://20somethingfinance.com/how-to-send-text-messages-sms-via-email-for-free
For me, I use my phone number @vtest.com, example: [email protected]. Note: you don't have to use SMS for this, you can enter any email address in the script to get email only alerts.
Once you have the correct email for your phone number and carrier, continue to the next section.
- Let's create the "checknode" script:
nano checknodePaste the following block of text and modify the first two variables for your node address and carrier's email
#!/bin/bash
# This is your node address, update this:
nodeAddress="0xc1f716d95de19bae6bf244756dce48da9a20271f"
# Your phone number in email form, update this:
phone="[email protected]"
# Delay in seconds between block checks. Recommended 60 or higher
delay=80
###################################################
##### DO NOT CHANGE ANYTHING BEYOND THIS LINE #####
###################################################
# Grab API data for node from TOMO Master
apiResults=$(curl -s "https://master.tomochain.com/api/candidates/$nodeAddress")
# Make sure there are no issues connecting to TOMO Master
exitCode=$?
if [ $exitCode != 0 ]
then
printf "Subject: ALERT\n\nCannot connect to TOMO Master, it may be down or the team is doing maintenance." | /usr/sbin/ssmtp $phone
exit $exitCode
fi
# Now start parsing TOMO Master results
nodeName=$(echo $apiResults | jq -r ".name")
nodeStatus=$(echo $apiResults | jq -r ".status")
isMasternode=$(echo $apiResults | jq -r ".isMasternode")
capacityNumber=$(echo $apiResults | jq -r ".capacityNumber")
# First thing to check: is the node Slashed or Proposed?
if [ "$isMasternode" != "true" ]
then
printf "Subject: ALERT\n\n${nodeName//[^[:ascii:]]/} is $nodeStatus | Capacity: $capacityNumber" | /usr/sbin/ssmtp $phone
exit 1
fi
# If the node status is OK, we need to make sure it's not stuck:
# grab current local block
first=$(docker exec $(docker ps -q -f "name=tomochain") tomo attach /tomochain/data/tomo.ipc --exec "eth.blockNumber")
# wait for the alloted delay set above
sleep $delay
# then grab the second block
second=$(docker exec $(docker ps -q -f "name=tomochain") tomo attach /tomochain/data/tomo.ipc --exec "eth.blockNumber")
# and compare the two to make sure they are not the same, if they are, send a text message alert
if [ "$first" -eq "$second" ]
then
printf "Subject: ALERT\n\n${nodeName//[^[:ascii:]]/} is stuck on block $second" | /usr/sbin/ssmtp $phone
exit 1
fi
exit 0- Make the script executable
chmod +x checknode- Create a cronjob that runs the checknode script every 5 minutes
crontab -eA text editor will load your crontab file. Paste the following on a new line then save and exit:
*/5 * * * * /home/tomo/checknodeNote: You will need to know the full path where you created the checknode script. If you followed this guide entirely, this will be in the home directory of your masternode user. Since my masternode user is "tomo" my script is in /home/tomo.