Introduce spl_kmem_caches command

[sdimitro]

= Motivation

/proc/spl/kmem/slab is a useful tool for debugging issues
in ZoL that are memory-related. Unfortunately, it is not
applicable to crash dumps and not very user friendly overall
(e.g. you get almost no information for spl caches that are
backed by Linux Slab caches).

Besides the above, the specific use-case within Delphix is a
replacement for ::kmastat that we had in MDB in the illumos
version of the product. The spl_kmem_caches command of this
commit is implemented to complement the slabs command and
cover the functionality of ::kmastat that we miss further.

= Patch

This commit introduces the spl directory in our code base
and implements spl_kmem_caches. By default the command
iterates over all the SPL caches and prints the following
statistics in human-readable form sorted by active_memory
used:

name                     entry_size active_objs active_memory                         source total_memory util
------------------------ ---------- ----------- ------------- ------------------------------ ------------ ----
zio_data_buf_131072          135680         760        98.3MB      zio_data_buf_131072[SPL ]      119.0MB   82
dnode_t                        1168       25533        28.4MB                  dnode_t[SLUB]       28.6MB   99
zfs_znode_cache                1104       23039        24.3MB          zfs_znode_cache[SLUB]       24.4MB   99
...

All the properties can be seen in the help message and the
user may specify the properties they are looking for with
the -o option (and also sort them by a specific field
using -s):

> spl_kmem_caches -h
usage: spl_kmem_caches [-h] [-H] [-o FIELDS] [-p] [-r] [-s FIELD] [-v]

optional arguments:
  -h, --help       show this help message and exit
  -H               do not display headers and separate fields by a single tab
                   (scripted mode)
  -o FIELDS        comma-separated list of fields to display
  -p               display numbers in parseable (exact) values
  -r, --recursive  recurse down children caches and print statistics
  -s FIELD         sort rows by FIELD
  -v               Print all statistics

FIELDS := address, name, flags, object_size, entry_size, slab_size,
objects_per_slab, entries_per_slab, slabs, active_slabs, active_memory,
total_memory, objs, active_objs, inactive_objs, source, util

If -o is not specified the default fields used are name, entry_size,
active_objs, active_memory, source, total_memory, util.

If the -s option is not specified and the command's output is not piped anywhere
then we sort by the following fields in order: active_memory, address, name. If
none of those exists in the field-set we sort by the first field specified in
the set.

> spl_kmem_caches -o "name,flags,slabs" -s slabs
name                                       flags slabs
------------------------ ----------------------- -----
zio_data_buf_131072         KMC_NODEBUG|KMC_VMEM   115
zio_buf_131072              KMC_NODEBUG|KMC_VMEM    25
zio_data_buf_40960          KMC_NODEBUG|KMC_VMEM    19
...

Besides a PrettyPrinter, the command is also a Locator which
means that the user can do things like this:

> spl_kmem_caches -s total_memory | head 1 | pp
name                entry_size active_objs active_memory                    source total_memory util
------------------- ---------- ----------- ------------- ------------------------- ------------ ----
zio_data_buf_131072     135680         760        98.3MB zio_data_buf_131072[SPL ]      119.0MB   82

A couple more options mimicking Illumos conventions are the
-p and -H options that output numbers to raw form and
skip printing the headers respectively. This is generally
useful for scripts or output that is to be processed later:

> spl_kmem_caches -H -p
zio_data_buf_131072	135680	760	103116800	zio_data_buf_131072[SPL ]	124825600	82
dnode_t	1168	25573	29869264	dnode_t[SLUB]	30005920	99
zfs_znode_cache	1104	23039	25435056	zfs_znode_cache[SLUB]	25548768	99

Note that for these command to fully work for SPL caches
that are backed by Linux Slab caches we need the appropriate
ZoL changes that enables us to track allocations for those
cases (see PR: openzfs/zfs#9474).
If the commit from that PR is not in the running system
then all of these caches will incorectly be reported empty.

Side-change:
As the spl_kmem_caches command is very close both in terms
of implementation and functionality with the slabs command
this patch also updates some of the columns and calculations
in the output of slabs, to make the commands consistent
with each other.

= Testing/Verification

I verified the values of all their fields in the following
ways:
[1] For the fields that are also visible in /proc/slabinfo,
I ensured that the values matched.
[2] For anything else I did the math and cross-referenced
all the values ensuring that they are consistent with
each other.

Besides the manual testing done shown in the output above I also
ensured the following error cases:

• ask for invalid field name

> spl_kmem_caches -o bogus
sdb: spl_kmem_caches: 'bogus' is not a valid field

• sort by invalid field name

sdb: spl_kmem_caches: invalid input: 'bogus' is not in field set (name, entry_size, active_objs, active_memory,
source, total_memory, util)

• attempt to sort by field that is not in the requested
  field set

> spl_kmem_caches -o "name,address" -s objs
sdb: spl_kmem_caches: invalid input: 'objs' is not in field set (name, address)

= Other Notes

The command only works with SLUB and until we have proper
lexing we still pass fields like this:
> spl_kmem_caches -o "name,address"

[sdimitro]

cc @ahrens

[prakashsurya]

LGTM