[sled-agent] Re-construct Zone management state within sled agent after reboot

[smklein]

Note: This issue tracks a follow-up from #686

Problem

Sled Agent should re-initialize in-memory state for managing Zones on boot, including:

• Identifying Omicron-owned zones
• Identifying their VNICs and other attached devices
• Creating in-memory structures to accurately represent their state (e.g. "RunningZone") such that they can be handled within the context of other Zones, and cleaned up accordingly.

Current workaround

As the Sled Agent "deletes all zones on initialization", we can punt on this issue slightly.

When this'll be a bigger problem

If we want to cope with Sled Agent itself crashing (without taking out all other software on the sled), this issue will become blocking.

[bnaecker]

This issue also needs to expand to include recovering the data links used for guest instances. These are currently the xde devices, plus VNICs over them (required by the Viona virtio-net driver).

We are currently deleting all of the guest Propolis zones, and also deleting these links. We'll need to recover and track them similar to the control VNICs used for communicating with Oxide services running in zones.

[askfongjojo]

Naive question: If sled-agent stops reaping propolis zones upon restart, would it be able to rely on info from the config files /etc/zones to re-construct the necessary zone info?

BRM42220081 # cat /etc/zones/oxz_propolis-server_8e00d49b-29e2-4b4f-8ab4-aefab6043c5b.xml 
<?xml version="1.0"?>
<!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
<!--
    DO NOT EDIT THIS FILE.  Use zonecfg(8) instead.
-->
<zone name="oxz_propolis-server_8e00d49b-29e2-4b4f-8ab4-aefab6043c5b" zonepath="/zone/oxz_propolis-server_8e00d49b-29e2-4b4f-8ab4-aefab6043c5b" autoboot="false" brand="omicron1" ip-type="exclusive">
  <device match="/dev/vmm/*"/>
  <device match="/dev/vmmctl"/>
  <device match="/dev/viona"/>
  <network physical="vopte1"/>
  <network physical="oxControlInstance0"/>
</zone>

The zone running state isn't there but putting a user VM at a stopped state would still be better than blowing it away as an initial solution to the problem.

In the future, we probably want to allow the user to configure the autoboot flag so that they can indicate whether the VM should be booted up by default if the last known running state is unavailable.

[jclulow]

Don't look in the configuration files, they're private; just use zoneadm list and zonecfg etc.

You would indeed be able to tell which zones are already running on the machine if sled agent is restarting without the system rebooting. If the system reboots, /etc is effectively restored to the pristine copy that ships in the ramdisk, which doesn't include any configured zones -- so the zone-level autoboot flag won't ever really have an effect on Gimlets.