propolis logs lost on reboot

[faithanalog]

While investigating an apparent crucible bug, we ended up with a sled crashed into kmdb. We had retrieved some propolis logs from that sled, but not all the ones we wanted.

Upon rebooting the sled, all of the propolis datasets appeared to have been deleted, and the propolis logs we wanted along with them.

The same zpools were present, so I do think that this was some system cleaning up after VMs that no longer existed after the sled crashed, but I'm not sure what exactly might have done this.

[smklein]

See:

omicron/sled-storage/src/dataset.rs

Lines 65 to 66 in 8d73079

	// Stores filesystems for zones
	ExpectedDataset::new(ZONE_DATASET).wipe(),

omicron/sled-storage/src/dataset.rs

Lines 110 to 111 in 8d73079

	// Identifies if the dataset should be deleted on boot
	wipe: bool,

omicron/sled-storage/src/dataset.rs

Lines 285 to 306 in 8d73079

	if dataset.wipe {
	match Zfs::get_oxide_value(name, "agent") {
	Ok(v) if &v == agent_local_value => {
	info!(log, "Skipping automatic wipe for dataset: {}", name);
	}
	Ok(_) | Err(_) => {
	info!(log, "Automatically destroying dataset: {}", name);
	Zfs::destroy_dataset(name).or_else(|err| {
	// If we can't find the dataset, that's fine -- it might
	// not have been formatted yet.
	if matches!(
	err.err,
	DestroyDatasetErrorVariant::NotFound
	) {
	Ok(())
	} else {
	Err(err)
	}
	})?;
	}
	}
	}

Unless you're reporting this as a regression, I think this is currently expected behavior. We are destroying all transient zone filesystems when the sled reboots, at the moment.

There are related issues to make the set of datasets less "implicit", and more "managed by Nexus". Of these, I'd say:

• Dataset accounting: VMM zone filesystems should be explicitly requested #6108
• Sled Agent must manage durable storage for configs, zones, explicitly #2888
• [sled-agent] Re-construct Zone management state within sled agent after reboot #725
• Sled agent tears down all running zones (including running VMs) if it crashes & restarts #2646

Are probably most relevant.

In particular:

• If we finish making Nexus aware of all U.2 dataset allocations, we can avoid this periodic "clear-on-reboot" behavior to garbage collect old instance filesystems.
• ... then we can make more significant progress on re-constructing instance state, rather than destroying them on boot.

[faithanalog]

This is all good background. I figured it was expected but didn't know anything about the mechanism.

The thing that bothers me about the behavior is mainly the loss of diagnostic data in the log files. My hope is that we could archive the logs from the zone filesystem somewhere before destroying the dataset (though- how would we manage the lifecycle of those logs after we do this?)

[smklein]

The Zone Bundler in sled-agent/src/zone_bundle.rs exists, and was created to take snapshots of unexpectedly dying zones. This may be a spot where we could re-use it.