[leo_statistics] some system metrics should be added for more reliability

[mocchira]

• recon_alloc:memory(usage), recon_alloc:memory(allocated) to confirm whether memory fragmentation occur on Erlang VM
• recon:scheduler_usage/1 to confirm the actual cpu usage is caused by Erlang schedulers or Erlang processes and so on
• recon:port_types/0 to confirm the number of Ports sorted by type(tcp|udp sockets, file, stdio and so on).
• The number of unlinked(and unmonitored) processes retrieved by the below Erlang code to confirm whether spawned processes leak

  length([P || P <- processes(), [{_,Ls},{_,Ms}] <- [process_info(P, [links,monitors])], []==Ls, []==Ms]).

• Use http://www.erlang.org/doc/man/erlang.html#system_monitor-2 to detect some suspicious behaviour such as Long GC time as early as possible.

[yosukehara]

Sample:

$ snmpwalk -v 2c -c public 127.0.0.1:4010 .1.3.6.1.4.1.35450.34
SNMPv2-SMI::enterprises.35450.34.1.0 = STRING: "[email protected]"
SNMPv2-SMI::enterprises.35450.34.2.0 = Gauge32: 298
SNMPv2-SMI::enterprises.35450.34.3.0 = Gauge32: 41916568
SNMPv2-SMI::enterprises.35450.34.4.0 = Gauge32: 23770028
SNMPv2-SMI::enterprises.35450.34.5.0 = Gauge32: 18152076
SNMPv2-SMI::enterprises.35450.34.6.0 = Gauge32: 5867308
SNMPv2-SMI::enterprises.35450.34.7.0 = Gauge32: 298
SNMPv2-SMI::enterprises.35450.34.8.0 = Gauge32: 41852943
SNMPv2-SMI::enterprises.35450.34.9.0 = Gauge32: 23774747
SNMPv2-SMI::enterprises.35450.34.10.0 = Gauge32: 18090933
SNMPv2-SMI::enterprises.35450.34.11.0 = Gauge32: 5872625
SNMPv2-SMI::enterprises.35450.34.12.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.13.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.14.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.15.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.16.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.17.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.18.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.19.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.20.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.21.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.22.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.23.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.24.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.31.0 = Gauge32: 68
SNMPv2-SMI::enterprises.35450.34.32.0 = Gauge32: 61958853
SNMPv2-SMI::enterprises.35450.34.33.0 = Gauge32: 68
SNMPv2-SMI::enterprises.35450.34.34.0 = Gauge32: 61635269

See also: leo-project/leo_statistics - Implement vm-related stats