V2: Agent log message details are not visible in the Fleet agent detail log view

[cmacknz]

In V2 the majority of the detail in agent log messages is now included in an component object that is not visible in any of the easily accessible views in Fleet. An example log message is:

{
  "log.level": "info",
  "@timestamp": "2022-12-15T19:44:45.271Z",
  "log.origin": {
    "file.name": "coordinator/coordinator.go",
    "file.line": 345
  },
  "message": "Existing component state changed",
  "component": {
    "id": "beat/metrics-monitoring",
    "state": "HEALTHY",
    "message": "Healthy: communicating with pid 81194",
    "inputs": [
      {
        "id": "beat/metrics-monitoring-metrics-monitoring-beats",
        "state": "HEALTHY",
        "message": "beat reloaded"
      }
    ],
    "output": {
      "id": "beat/metrics-monitoring",
      "state": "CONFIGURING",
      "message": "reloading output component"
    }
  },
  "ecs.version": "1.6.0"
}

In Fleet this is presented only as "Existing component state changed" with very little detail:

Following the "Open in Logs" link presents slightly more detail but still omits most of the useful information:

[cmacknz]

@joshdover what controls the visibility of the message fields here? Is there a mapping we need to update on the Fleet side?

[joshdover]

The mappings are populated by the elastic_agent package's field definitions: https://github.com/elastic/integrations/tree/main/packages/elastic_agent/data_stream/elastic_agent_logs/fields

This would need to be updated for the components key and then a new package version needs to be released and the version bundled with kibana needs to be updated here (and make sure it's backported to 8.6): https://github.com/elastic/kibana/blob/fa9094366c63aa371b7010a19e5ba5eb99b74236/fleet_packages.json#L28

That said I do think we should make the message field more useful as well as this is the only field presented in the Logs and Fleet UI. Maybe a format like:

[${component.id}][${component.state}] Existing component state changed: ${component.message}, inputs: ${component.inputs.map(i => i.state).join(",")}, output: ${component.output.state}

[cmacknz]

Agreed, just changing the log message here would help. We are using the logger.With pattern to attach the component log object everywhere so it might be a bit tricky to find every instance of this in the agent code, but at least for this state changed message we can change it since it is the most useful message in the logs usually.

[joshdover]

Testing my fix is blocked on elastic/beats#34093

[cmacknz]

Blake also made some changes to address this in #2001