[sled agent] Fixes to enable the reboot after RSS initialization

[smklein]

This PR consists of changes to the Sled Agent to allow internal services to re-launch successfully after a reboot.

This was tested with the following script, on an initialized system running with SoftNPU:

# Emulate a "sled shutdown"
svcadm disable sled-agent

# Kill and restart softnpu, to "empty out switch state"
zlogin softnpu pkill softnpu
zlogin softnpu pgrep softnpu || {
    zlogin softnpu 'RUST_LOG=debug /stuff/softnpu /stuff/softnpu.toml &> /softnpu.log &'
}

# Restart the sled agent. Observe that it still tears down all running zones on boot,
# but it successfully re-initializes all sleds which are described by:
# /pool/int/<UUID>/config/services.toml
svcadm enable sled-agent

Before this PR

• Sled Agent was capable of booting and initializing via RSS
• However, after a reboot, the Sled Agent did not successfully re-initialize all zones that had been running before reboot. It attempted to bring up the switch zone, failed, and would not be able to continue initializing other services

This PR

• Fixes the routing issue described by Reboot: Switch Zone unhappy with route - missing underlay address #3461: Zone initialization only attempts to add a route to the "sled agent underlay address" as a gateway iff (1) the underlay is up, and (2) the zone-being-launched has an address on the same subnet.
• Delays launching all "Sled Agent services" until after the sled agent has an underlay address. This makes it more likely for services dependent on external connectivity (e.g., NTP, external DNS, Nexus) to initialize correctly.
• The bootstrap agent, which may send a start_rack_initialize request if and RSS configuration file is included, no longer throws an error if RSS has already been initialized.

Fixes #3461
Fixes #3106
Part of #725

[andrewjstone]

Sean this is awesome. I'm very excited to see this go in. Just one question for my understanding purposes.

[andrewjstone]

Will the switch zone eventually get an underlay address and add a route?

[smklein]

Good question, and the answer is: Yes, it will.

When we first boot up, the BootstrapAgent is responsible for "hardware monitoring", which starts the switch zone iff the tofino is detected:

omicron/sled-agent/src/bootstrap/agent.rs

Lines 397 to 404 in 3e020c9

	let hardware_monitor = Self::hardware_monitor(
	&ba_log,
	&config.link,
	&sled_config,
	global_zone_bootstrap_link_local_address,
	storage_key_requester.clone(),
	)
	.await?;

This calls HardwareMonitor::new, which itself has a reference to the ServiceManager structure:

omicron/sled-agent/src/bootstrap/hardware.rs

Line 211 in 92258b4

let service_manager = ServiceManager::new(

(Note, this is the same ServiceManager structure that the SledAgent will later use to manage services when we've fully booted)

The bootstrap's HardwareManager spawns a tokio task which monitors hardware for the Tofino, and in response, launches the switch zone:

omicron/sled-agent/src/bootstrap/hardware.rs

Lines 72 to 79 in 92258b4

	let baseboard = self.hardware.baseboard();
	let switch_zone_ip = None;
	if let Err(e) = self.services.activate_switch(
	switch_zone_ip,
	baseboard,
	).await {
	warn!(self.log, "Failed to activate switch: {e}");
	}

It's important to note -- the let switch_zone_ip = None line is critical. If the bootstrap agent is launching the switch zone, it might be doing so before the underlay is up.

Later on, when the BootstrapAgent tries to launch the SledAgent. It stops monitoring hardware at this point.

omicron/sled-agent/src/bootstrap/agent.rs

Lines 526 to 527 in 92258b4

	// Stop the bootstrap agent from monitoring for hardware, and
	// pass control of service management to the sled agent.

And when the SledAgent starts, it does hardware monitoring for itself:

omicron/sled-agent/src/sled_agent.rs

Lines 336 to 340 in 92258b4

	// Begin monitoring the underlying hardware, and reacting to changes.
	let sa = sled_agent.clone();
	tokio::spawn(async move {
	sa.hardware_monitor_task(log).await;
	});

If the switch zone already launched, the SledAgent will call activate_switch with the underlay IP address:

omicron/sled-agent/src/sled_agent.rs

Lines 353 to 365 in 92258b4

	let scrimlet = self.inner.hardware.is_scrimlet_driver_loaded();
	if scrimlet {
	let baseboard = self.inner.hardware.baseboard();
	let switch_zone_ip = Some(self.inner.switch_zone_ip());
	if let Err(e) = self
	.inner
	.services
	.activate_switch(switch_zone_ip, baseboard)
	.await
	{
	warn!(log, "Failed to activate switch: {e}");
	}

(this is poking at the same ServiceManager used by the bootstrap agent earlier!)

This case is handled explicitly: If the switch zone is running, and needs to be refreshed to add an underlay address and route. It's the same path we use during cold boot / RSS!

omicron/sled-agent/src/services.rs

Lines 2742 to 2748 in 2d46f8c

	(SledLocalZone::Running { request, zone }, Some(new_request))
	if request.addresses != new_request.addresses =>
	{
	// If the switch zone is running but we have new addresses, it
	// means we're moving from the bootstrap to the underlay
	// network. We need to add an underlay address and route in the
	// switch zone, so dendrite can communicate with nexus.

[andrewjstone]

Aha! It was those last two steps I was missing. Thanks for pointing this out!