[sled agent] Creating zones for Propolis Instances leaks datasets

[smklein]

Our model of "zone management" for Propolis servers expects that:

• When an instance is running, we'll create + boot a zone for it
• When an instance is stopped, all resources allocated for that zone should be freed

As a result of "testing instance creation + stop + delete" in a loop, however, I'm seeing zone datasets be created for each of these services without being destroyed:

# Note: This is after create + stop + delete in a loop. I would only expect at most one dataset to exist at a time!
$ zfs list
...
rpool/zone/oxz_propolis-server_50cbb8a9-a202-448a-84c2-be37866ccbde                               66K   746G    66K  /zone/oxz_propolis-server_50cbb8a9-a202-448a-84c2-be37866ccbde
rpool/zone/oxz_propolis-server_551e3e8e-9f95-4cb0-b782-c705009fceda                              463M   746G   463M  /zone/oxz_propolis-server_551e3e8e-9f95-4cb0-b782-c705009fceda
rpool/zone/oxz_propolis-server_555ca2fb-275d-462d-93b2-df4d41426e42                               71K   746G    71K  /zone/oxz_propolis-server_555ca2fb-275d-462d-93b2-df4d41426e42
rpool/zone/oxz_propolis-server_8e37d3be-1389-4b1d-be71-d028ceedde50                               79K   746G    79K  /zone/oxz_propolis-server_8e37d3be-1389-4b1d-be71-d028ceedde50
rpool/zone/oxz_propolis-server_9eff450f-38c5-46a0-9e49-2b9f489f7cfc                               86K   746G    86K  /zone/oxz_propolis-server_9eff450f-38c5-46a0-9e49-2b9f489f7cfc
rpool/zone/oxz_propolis-server_ea99d0d5-3d45-46b0-8e9b-fe0ff1877ffd                               63K   746G    63K  /zone/oxz_propolis-server_ea99d0d5-3d45-46b0-8e9b-fe0ff1877ffd
rpool/zone/oxz_propolis-server_eafdbfb5-c162-487e-8faa-3e89106fe648                               66K   746G    66K  /zone/oxz_propolis-server_eafdbfb5-c162-487e-8faa-3e89106fe648

[smklein]

If you're feeling brave, the following command deletes all of these:

zfs list -p | cut -d' ' -f1 | grep propolis-server | xargs -I {} pfexec zfs destroy {}

However, ideally, this should be the responsibility of the Sled Agent itself.

[leftwo]

Crucible appears to suffer the same fate