Skip to content

Commit

Permalink
[gateway] ingest sensor measurements from SPs into oximeter (#6354)
Browse files Browse the repository at this point in the history 
This branch adds code to the Management Gateway Service for periodically
polling sensor measurements from SPs and emitting it to Oximeter. In
particular, this consists of:

- a task for managing the metrics endpoint, waiting until MGS knows its
  underlay network address to bind the endpoint and register it with the
  control plane,
- tasks for polling sensor measurements from each individual SP that MGS
  knows about,
- a task that waits until SP discovery has completed and the rack ID to
  be known, and then spawns a poller task for every discovered SP slot

The SP poller tasks send samples to the Oximeter producer endpoint using
a `tokio::sync::broadcast` channel, which I've chosen primarily because
it can be used as a bounded ring buffer that actually overwrites the
*oldest* value when the buffer is full. This mostway, we use a bounded
amount of memory for samples, but prioritize the most recent samples if
we have to throw anything away because Oximeter hasn't come along to
collect them recently.

The poller tasks cache the component inventory and identifying
information from the SP, so that we don't have to re-read all this data
from the SP on every poll. While MGS, running on a host, would probably
be fine with doing this, it seems better to avoid making the SP do
unnecessary work at a 1Hz poll frequency, especially when *both* switch
zones are polling them. Instead, every time we poll sensor data from an
SP, we first ask it for its current state, and only invalidate our
cached understanding of the SP when the state changes. This way, if a SP
starts reporting new metrics due to a firmware update, or gets replaced
with a different chassis with a new serial number, revision, etc, we
won't continue to report metrics for stale targets, but we don't have to
reload all of that once per second. To detect scenarios where the SP's
state and/or identity has changed in the midst of polling its sensors
(which may result in mislabeled metrics), we check whether the SP's
state at the end of the poll matches its state at the beginning, and if
it's not, we poll again immediately with its new identity. 

At present, the timestamps for these metric samples is generated by MGS
--- it's the time when MGS received the sensor data from the SP, as MGS
understands it. Because we don't currently collect data that was
recorded prior to the switch zone coming up, we don't need to worry
about figuring out timestamps for data recorded by the SP prior to the
existence of a wall clock. Figuring out the SP/MGS timebase
synchronization is probably a lot of additional work, although it would
be nice to do in the future. At present, [metrics emitted by sled-agent
prior to NTP sync will also be from 1987][1], so I think it's fine to do
something similar here, especially because the potential solutions to
that [also have their fair share of tradeoffs][2].

The new metrics use a schema in
`oximeter/oximeter/schema/hardware-component.toml`. The target of these
metrics is a `hardware_component` that includes:

- the rack ID and the identity of the MGS instance that collected the
  metric,
- information identifying the chassis[^1] and of the SP that recorded
them (its serial number, model number, revision, and whether it's a
switch, a sled, or a power shelf),
- the SP's Hubris archive version (since the reported sensor data may
  change in future firmware releases)
- the SP's ID for the hardware component (e.g. "dev-7"), the kind of
  device (e.g. "tmp117", "max5970"), and the humman-readable description
  (e.g. "Southeast temperature sensor", "U.2 Sharkfin A hot swap
  controller", etc.) reported by the SP

Each kind of sensor reading has an individual metric
(`hardware_component:temperature`, `hardware_component:current`,
`hardware_component:voltage`, and so on). These metrics are labeled with
the SP-reported name of the individual sensor measurement channel. For
instance, a MAX5970 hotswap controller on sharkfin will have a voltage
and current metric named "V12_U2A_A0" for the 12V rail, and a voltage
and current metric named "V3P3_U2A_A0" for the 3.3V rail. Finally, a
`hardware_component:sensor_errors` metric records sensor errors reported
by the SP, labeled with the sensor name, what kind of sensor it is, and
a string representation of the error.

[1]:
    #6354 (comment)
[2]:
    #6354 (comment)

[^1]: I'm using "chassis" as a generic term to refer to "switch, sled,
    or power shelf".
  • Loading branch information
@hawkw
hawkw authored Aug 24, 2024
1 parent 31ea57e commit 5afa0de
Show file tree
Hide file tree
Showing 28 changed files with 1,990 additions and 33 deletions.
4 changes: 4 additions & 0 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions clients/nexus-client/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ impl From<omicron_common::api::internal::nexus::ProducerKind>
fn from(kind: omicron_common::api::internal::nexus::ProducerKind) -> Self {
use omicron_common::api::internal::nexus::ProducerKind;
match kind {
ProducerKind::ManagementGateway => Self::ManagementGateway,
ProducerKind::SledAgent => Self::SledAgent,
ProducerKind::Service => Self::Service,
ProducerKind::Instance => Self::Instance,
Expand Down Expand Up @@ -390,6 +391,9 @@ impl From<types::ProducerKind>
fn from(kind: types::ProducerKind) -> Self {
use omicron_common::api::internal::nexus::ProducerKind;
match kind {
types::ProducerKind::ManagementGateway => {
ProducerKind::ManagementGateway
}
types::ProducerKind::SledAgent => ProducerKind::SledAgent,
types::ProducerKind::Instance => ProducerKind::Instance,
types::ProducerKind::Service => ProducerKind::Service,
Expand Down
1 change: 1 addition & 0 deletions clients/oximeter-client/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ impl From<omicron_common::api::internal::nexus::ProducerKind>
fn from(kind: omicron_common::api::internal::nexus::ProducerKind) -> Self {
use omicron_common::api::internal::nexus;
match kind {
nexus::ProducerKind::ManagementGateway => Self::ManagementGateway,
nexus::ProducerKind::Service => Self::Service,
nexus::ProducerKind::SledAgent => Self::SledAgent,
nexus::ProducerKind::Instance => Self::Instance,
Expand Down
2 changes: 2 additions & 0 deletions common/src/api/internal/nexus.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@ pub enum ProducerKind {
Service,
/// The producer is a Propolis VMM managing a guest instance.
Instance,
/// The producer is a management gateway service.
ManagementGateway,
}

/// Information announced by a metric server, used so that clients can contact it and collect
Expand Down
1 change: 1 addition & 0 deletions dev-tools/mgs-dev/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ futures.workspace = true
gateway-messages.workspace = true
gateway-test-utils.workspace = true
libc.workspace = true
omicron-gateway.workspace = true
omicron-workspace-hack.workspace = true
signal-hook-tokio.workspace = true
tokio.workspace = true
24 changes: 22 additions & 2 deletions dev-tools/mgs-dev/src/main.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ use clap::{Args, Parser, Subcommand};
use futures::StreamExt;
use libc::SIGINT;
use signal_hook_tokio::Signals;
use std::net::SocketAddr;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
Expand Down Expand Up @@ -36,7 +37,12 @@ enum MgsDevCmd {
}

#[derive(Clone, Debug, Args)]
struct MgsRunArgs {}
struct MgsRunArgs {
/// Override the address of the Nexus instance to use when registering the
/// Oximeter producer.
#[clap(long)]
nexus_address: Option<SocketAddr>,
}

impl MgsRunArgs {
async fn exec(&self) -> Result<(), anyhow::Error> {
Expand All @@ -46,9 +52,23 @@ impl MgsRunArgs {
let mut signal_stream = signals.fuse();

println!("mgs-dev: setting up MGS ... ");
let gwtestctx = gateway_test_utils::setup::test_setup(
let (mut mgs_config, sp_sim_config) =
gateway_test_utils::setup::load_test_config();
if let Some(addr) = self.nexus_address {
mgs_config.metrics =
Some(gateway_test_utils::setup::MetricsConfig {
disabled: false,
dev_nexus_address: Some(addr),
dev_bind_loopback: true,
});
}

let gwtestctx = gateway_test_utils::setup::test_setup_with_config(
"mgs-dev",
gateway_messages::SpPort::One,
mgs_config,
&sp_sim_config,
None,
)
.await;
println!("mgs-dev: MGS is running.");
Expand Down
25 changes: 20 additions & 5 deletions dev-tools/omdb/tests/successes.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,16 @@ SP DETAILS: type "Sled" slot 0

COMPONENTS

NAME DESCRIPTION DEVICE PRESENCE SERIAL
sp3-host-cpu FAKE host cpu sp3-host-cpu Present None
dev-0 FAKE temperature sensor fake-tmp-sensor Failed None
NAME DESCRIPTION DEVICE PRESENCE SERIAL
sp3-host-cpu FAKE host cpu sp3-host-cpu Present None
dev-0 FAKE temperature sensor fake-tmp-sensor Failed None
dev-1 FAKE temperature sensor tmp117 Present None
dev-2 FAKE Southeast temperature sensor tmp117 Present None
dev-6 FAKE U.2 Sharkfin A VPD at24csw080 Present None
dev-7 FAKE U.2 Sharkfin A hot swap controller max5970 Present None
dev-8 FAKE U.2 A NVMe Basic Management Command nvme_bmc Present None
dev-39 FAKE T6 temperature sensor tmp451 Present None
dev-53 FAKE Fan controller max31790 Present None

CABOOSES: none found

Expand All @@ -167,8 +174,16 @@ SP DETAILS: type "Sled" slot 1

COMPONENTS

NAME DESCRIPTION DEVICE PRESENCE SERIAL
sp3-host-cpu FAKE host cpu sp3-host-cpu Present None
NAME DESCRIPTION DEVICE PRESENCE SERIAL
sp3-host-cpu FAKE host cpu sp3-host-cpu Present None
dev-0 FAKE temperature sensor tmp117 Present None
dev-1 FAKE temperature sensor tmp117 Present None
dev-2 FAKE Southeast temperature sensor tmp117 Present None
dev-6 FAKE U.2 Sharkfin A VPD at24csw080 Present None
dev-7 FAKE U.2 Sharkfin A hot swap controller max5970 Present None
dev-8 FAKE U.2 A NVMe Basic Management Command nvme_bmc Present None
dev-39 FAKE T6 temperature sensor tmp451 Present None
dev-53 FAKE Fan controller max31790 Present None

CABOOSES: none found

Expand Down
9 changes: 9 additions & 0 deletions gateway-test-utils/configs/config.test.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,15 @@ addr = "[::1]:0"
ignition-target = 3
location = { switch0 = ["sled", 1], switch1 = ["sled", 1] }

#
# Configuration for SP sensor metrics polling
#
[metrics]
# Allow the Oximeter metrics endpoint to bind on the loopback IP. This is
# useful in local testing and development, when the gateway service is not
# given a "real" underlay network IP.
dev_bind_loopback = true

#
# NOTE: for the test suite, if mode = "file", the file path MUST be the sentinel
# string "UNUSED". The actual path will be generated by the test suite for each
Expand Down
166 changes: 166 additions & 0 deletions gateway-test-utils/configs/sp_sim_config.test.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,19 @@ device = "fake-tmp-sensor"
description = "FAKE temperature sensor 1"
capabilities = 0x2
presence = "Present"
sensors = [
{name = "Southwest", kind = "Temperature", last_data.value = 41.7890625, last_data.timestamp = 1234 },
]

[[simulated_sps.sidecar.components]]
id = "dev-1"
device = "fake-tmp-sensor"
description = "FAKE temperature sensor 2"
capabilities = 0x2
presence = "Failed"
sensors = [
{ name = "South", kind = "Temperature", last_error.value = "DeviceError", last_error.timestamp = 1234 },
]

[[simulated_sps.sidecar]]
multicast_addr = "::1"
Expand Down Expand Up @@ -56,6 +62,82 @@ device = "fake-tmp-sensor"
description = "FAKE temperature sensor"
capabilities = 0x2
presence = "Failed"
sensors = [
{ name = "Southwest", kind = "Temperature", last_error.value = "DeviceError", last_error.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-1"
device = "tmp117"
description = "FAKE temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "South", kind = "Temperature", last_data.value = 42.5625, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-2"
device = "tmp117"
description = "FAKE Southeast temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "Southeast", kind = "Temperature", last_data.value = 41.570313, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-6"
device = "at24csw080"
description = "FAKE U.2 Sharkfin A VPD"
capabilities = 0x0
presence = "Present"

[[simulated_sps.gimlet.components]]
id = "dev-7"
device = "max5970"
description = "FAKE U.2 Sharkfin A hot swap controller"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "V12_U2A_A0", kind = "Current", last_data.value = 0.45898438, last_data.timestamp = 1234 },
{ name = "V3P3_U2A_A0", kind = "Current", last_data.value = 0.024414063, last_data.timestamp = 1234 },
{ name = "V12_U2A_A0", kind = "Voltage", last_data.value = 12.03125, last_data.timestamp = 1234 },
{ name = "V3P3_U2A_A0", kind = "Voltage", last_data.value = 3.328125, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-8"
device = "nvme_bmc"
description = "FAKE U.2 A NVMe Basic Management Command"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "U2_N0", kind = "Temperature", last_data.value = 56.0, last_data.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-39"
device = "tmp451"
description = "FAKE T6 temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "t6", kind = "Temperature", last_data.value = 70.625, last_data.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-53"
device = "max31790"
description = "FAKE Fan controller"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "Southeast", kind = "Speed", last_data.value = 2607.0, last_data.timestamp = 1234 },
{ name = "Northeast", kind = "Speed", last_data.value = 2476.0, last_data.timestamp = 1234 },
{ name = "South", kind = "Speed", last_data.value = 2553.0, last_data.timestamp = 1234 },
{ name = "North", kind = "Speed", last_data.value = 2265.0, last_data.timestamp = 1234 },
{ name = "Southwest", kind = "Speed", last_data.value = 2649.0, last_data.timestamp = 1234 },
{ name = "Northwest", kind = "Speed", last_data.value = 2275.0, last_data.timestamp = 1234 },
]


[[simulated_sps.gimlet]]
multicast_addr = "::1"
Expand All @@ -72,6 +154,90 @@ capabilities = 0
presence = "Present"
serial_console = "[::1]:0"


[[simulated_sps.gimlet.components]]
id = "dev-0"
device = "tmp117"
description = "FAKE temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "Southwest", kind = "Temperature", last_data.value = 41.3629, last_data.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-1"
device = "tmp117"
description = "FAKE temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "South", kind = "Temperature", last_data.value = 42.5625, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-2"
device = "tmp117"
description = "FAKE Southeast temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "Southeast", kind = "Temperature", last_data.value = 41.570313, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-6"
device = "at24csw080"
description = "FAKE U.2 Sharkfin A VPD"
capabilities = 0x0
presence = "Present"

[[simulated_sps.gimlet.components]]
id = "dev-7"
device = "max5970"
description = "FAKE U.2 Sharkfin A hot swap controller"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "V12_U2A_A0", kind = "Current", last_data.value = 0.41893438, last_data.timestamp = 1234 },
{ name = "V3P3_U2A_A0", kind = "Current", last_data.value = 0.025614603, last_data.timestamp = 1234 },
{ name = "V12_U2A_A0", kind = "Voltage", last_data.value = 12.02914, last_data.timestamp = 1234 },
{ name = "V3P3_U2A_A0", kind = "Voltage", last_data.value = 3.2618, last_data.timestamp = 1234 },
]

[[simulated_sps.gimlet.components]]
id = "dev-8"
device = "nvme_bmc"
description = "FAKE U.2 A NVMe Basic Management Command"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "U2_N0", kind = "Temperature", last_data.value = 56.0, last_data.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-39"
device = "tmp451"
description = "FAKE T6 temperature sensor"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "t6", kind = "Temperature", last_data.value = 70.625, last_data.timestamp = 1234 },
]
[[simulated_sps.gimlet.components]]
id = "dev-53"
device = "max31790"
description = "FAKE Fan controller"
capabilities = 0x2
presence = "Present"
sensors = [
{ name = "Southeast", kind = "Speed", last_data.value = 2510.0, last_data.timestamp = 1234 },
{ name = "Northeast", kind = "Speed", last_data.value = 2390.0, last_data.timestamp = 1234 },
{ name = "South", kind = "Speed", last_data.value = 2467.0, last_data.timestamp = 1234 },
{ name = "North", kind = "Speed", last_data.value = 2195.0, last_data.timestamp = 1234 },
{ name = "Southwest", kind = "Speed", last_data.value = 2680.0, last_data.timestamp = 1234 },
{ name = "Northwest", kind = "Speed", last_data.value = 2212.0, last_data.timestamp = 1234 },
]


#
# NOTE: for the test suite, the [log] section is ignored; sp-sim logs are rolled
# into the gateway logfile.
Expand Down
Loading

0 comments on commit 5afa0de

Please sign in to comment.