APPSERV-47 Adds Custom Watches to Monitoring Console #4463

jbee · 2020-01-30T11:26:22Z

Summary

Adds custom watches to client. In contrast to watches collected from MonitoringWatchSources a custom watch is a user defined Watch instance. The monitoring client is used to compose such a watch. This can be done in the new page Watches that also lists all watches and allows to edit, delete or disable and enable them.

A disabled watch is simply not actively evaluating data, it is ignored but it will show in the graphs as a outline box instead of a filled one. All watches can be disabled and enabled. This is stored in the added configuration so that the disabled state can be recreated on server startup.

Only custom watches can be delete. Collected (or programmatic) watches cannot be deleted as they originate from the server's code.

Custom watches can be edited. Programmatic watches can be duplicated, which changes the name of the duplicate to Copy of {Name}. So this becomes another watch but allows to use the programmatic watch as a template.

To recreate the custom watches on server-startup the watches need to be stored in some form.
I decided to write custom to and from JSON methods for the affected classes. This will allow to put in the proper code that can handle changes to the data structure. The strategy here is to be able to handle missing fields (that would be added) and ignore fields (that would be removed).
The conversion is covered with unit tests.

On server side:

adds monitoring console configuration to store disabled watches as well as custom watch data
REST and service API to enable/disable watches
distinction between custom watches and programmatic watches
watch states are shared when deriving a watch
REST and service API to create custom watches
REST and service API to delete custom watches
WatchData REST API class is not writable to receive custom watch data send by client

On client side:

adds a Watches configuration page that lists existing watches, allows to edit, delete or disable/enable them.
extracts model constants to a separate file for better readability and clarity
extracts a controller file that encapsulation the client-server communication on client side for better readability and clarity
fixes/updates formatting of watch conditions (e.g. in alert tables) and watch lists
uses payara design system colors with better contrast
fixes alignment of indicator (text message above graph)
uses configured colours for status messages
now also triggers status messages from alerts (red=critical, amber=alerting)
adds new colour defaults for error and missing status
use of const instead of let (TLDR; const should be default, let only when variable is actually changed, var is basically never needed or better suited in our "non-legacy" codebase)
watch indicator at right side of the graph now shows the indicator as outline if the watch is disabled
Setting input with unit now can use a function to provide a changing unit.

`WriteableView` and `ProtectedList`

Tried to improve it to make it more usable. Code also could do with some cleanup.

added generics
added missing @Override annoations
removed unnecessary else nesting
made static helpers static
added some List methods that would otherwise throw a UnsupportedOperationException on the basis of existing method (so no actual transaction management was needed)

As mentioned some of the existing methods of the ProtectedList still show some odd behaviour.
When using set(int, E) the value at the index would be null next time I loaded the list.

Tests

Unit tests were added to verify the correctness of the to/from JSON conversion of the Watch class and all classes involved.

General information can be taken from the documentation https://github.com/payara/Payara-Server-Documentation/pull/705.

The custom watches feature was tested manually following the steps below:

General Setup:

build, install and start the server
use set-monitoring-console-configuration --enabled=true to deploy MC
open MC at http://localhost:8080/monitoring-console/
make sure browser cache for JS/CSS is cleared for MC's domain
check that following pages do exist: Watches (if not most likely a browser cache issue - or get in touch)

Testing Disabling Existing Watches

Open page Watches in MC
click checkbox of one of the listed watches to disable it, e.g. Metric Collection Duration
for Metric Collection Duration watch open page Monitoring and check the watch indicator on right side of the graph is now drawn as outline instead of filled
restart the server, e.g. using asadmin commands
refresh page Watches in MC and check disabled watch is still disabled
enable the watch again
for Metric Collection Duration watch open page Monitoring again and check the indicator is now drawn filled

Testing Disabling Custom Watches

Open page Watches in MC
create a new watch named Heap Check by checking Unhealthy, filling in: If ns:jvm HeapUsage in Percent is > (some value clearly below your current heap usage, see Core page), and press Save or Update
goto Core page and check that the graph shows an alert because heap usage is above the watch level
go back to Watches page and disable the created watch by clicking the checkbox on the left or using the cogs menu
go back to Core and check that alert is gone (and that indicator is now as outline)
go to Alerts page and check the alert is shown in Past Unacknowledged Alerts
go to Watches page again an enable the watch once more, check on Core page that an alert is triggered again

Testing Custom Watches

create a custom watch as given in steps for Testing Disabling Custom Watches
restart the server
open Watches page and check the custom watch created is still there

Testing Listing of collected Watches

open Watches page of MC, with default settings two watches should be listed
open admin console and browse to Configurations => server-config => HealthCheck_ tab CPU Usage and check Enabled
go back to Watches page and check a new watch CPU Usage is listed now

Testing new admin command parameters are hidden

run:

asadmin> set-monitoring-console-configuration --help
NAME
     set-monitoring-console-configuration

SYNOPSIS
     Usage: set-monitoring-console-configuration [--enabled=true|
     false] 

OPTIONS
     --enabled

Command set-monitoring-console-configuration executed successfully.

…tured watch list and editor

…tches have no effect on visuals

…tatus, colors and triggers linked to alert system

…config

…artup

jbee · 2020-01-30T11:34:07Z

...ore/src/main/java/fish/payara/monitoring/admin/SetMonitoringConsoleConfigurationCommand.java

+
+    @Param(optional = true, alias = "remove-watch")
+    private String removeWatch;
+


NB. I opted for a parameter for each action. This could also be done by adding a general parameter watch for the name and data for the JSON and another parameter action or so. But I found this easier to use. On the other hand these are not really intended to be used manually. They mainly exist so that the service can run them to update the config. Manual use is possible but will take a restart to take effect.

If they're not for manual use you can prepend them with an underscore (or two underscores, can't remember) to hide them as options on the command line (but not from REST).

jbee · 2020-01-30T11:41:13Z