sled agent sometimes publishes the Destroyed instance state when it should publish Stopped

[gjcolombo]

Nexus reserves the "Destroyed" instance state for instances that have actually been deleted. If sled agent pushes an instance state update (or returns an instance runtime state) to Nexus that bears the "Destroyed" state, things get messy, because Nexus won't let you destroy an instance that already appears to be destroyed.

The two places where this appears to happen are

• put_state on an instance that has been registered (so it's known to sled agent's instance manager) but not started yet
• an explicit call to terminate an instance (used when explicitly unregistering an instance without going through its Propolis; Nexus requests this in some saga unwind paths)

[gjcolombo]

This can cause instances to leak into the "Destroyed" state relatively easily. The most straightforward way to achieve this is to ask Nexus to stop an instance that's in the process of stopping:

1. T1: Propolis reaches its internal Destroyed state and publishes this back up to sled agent
2. T2: Parallel stop request arrives, reaches instance_manager::ensure_state, and successfully looks up the instance, obtaining a reference to it
3. T1: The Propolis's state monitor task takes the instance lock, tears down the zone, clears the instance's running_state, and releases the lock
4. T2: The stop request takes the instance lock, sees that there's no zone, and erroneously moves the instance to the Destroyed state and returns this state to Nexus

I can hit this within just a few seconds of running a stress test that I wrote for the occasion.

Switching the sled agent calls that transition instances to Destroyed to instead try to transition them to Stopped seems to mitigate the problem.