Tune container.image.* fields to follow OCI spec

[ChrsMark]

Summary

We need ensure that ECS is aligned with Otel's semantic conventions. In open-telemetry/semantic-conventions#48 it's been discussed how the container.image.* fields will be aligned with the OCI spec.
This is proposes the same that has been proposed at open-telemetry/semantic-conventions#48 (comment).

Also related to open-telemetry/semantic-conventions#72.

Motivation:

The motivation for this is to achieve allignment with Otel semantic conventions and more importantly with the OCI spec.

Detailed Design:

Copying the proposal that affects ECS from open-telemetry/semantic-conventions#48 (comment):

1. Add container.image.id: string
2. convert container.image.hash.all: string[] to container.image.digest: string[]

container.image.digest: string[]
container.image.id: string

Docker ✅

This is aligned with Docker where an inspect of an image would give sth like the following:

{
  "Id": "sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710",
  "RepoTags": [
  "example:1.0",
  "example:latest",
  "example:stable",
  "internal.registry.example.com:5000/example:1.0"
  ],
  "RepoDigests": [
  "example@sha256:afcc7f1ac1b49db317a7196c902e61c6c3c4607d63599ee1a82d702d249a0ccb",
  "internal.registry.example.com:5000/example@sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578"
  ],
  ...
}

OCI manifest ✅

This is also aligned with https://github.com/opencontainers/image-spec/blob/main/manifest.md which indicates that there are multiple digests out of the many layers an image is built.

k8s ✅

At the same time it's also aligned with what k8s report as I mention in my examples at open-telemetry/semantic-conventions#48 (comment) and according to the k8s docs:

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#containerstatus-v1-core
• https://kubernetes.io/docs/concepts/containers/images/
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#container-v1-core

Example:

Containers:
  etcd:
    Container ID:  containerd://6c531cfb94115489f86783cc6b11712f604a7d82c3b4693327ea1106e5455d13
    Image:         registry.k8s.io/etcd:3.5.7-0
    Image ID:      sha256:86b6af7dd652c1b38118be1c338e9354b33469e69a218f7e290a0ca5304ad681

CRI ✅

The the Container Runtime Interface (CRI) of k8s also (as expected :)) follow the OCI spec and hence this proposal is alligned with this one too: https://github.com/kubernetes/cri-api/blob/c75ef5b473bbe2d0a4fc92f82235efd665ea8e9f/pkg/apis/runtime/v1/api.proto#L1234-L1238

Example:

ID: sha256:431553d4c70b50f59fa9a57a1136e5833d2700210cda6c5f67770c982a8307f2
RepoTags: registry.k8s.io/kube-proxy:v1.27.1
RepoDigests: docker.io/library/import-2023-05-12@sha256:a9d9b674a55b137b1f10bbcd7f0e43b7568d20c8df5bf969d1bb8a37f3d365db
Size: 72702973

@norrietaylor based on our conversation this change of the container.image.hash.all field would affect your team. Could you please bring in someone from the team to validate this proposal and agree on the changes?

cc: @mlunadia @AlexanderWert

[ChrsMark]

open-telemetry/semantic-conventions#159 have been merged. We need to adjust ECS to at least:

1. container.image.tag -> container.image.tags
2. container.image.hash.all -> container.image.repo_digests

and optionally add container.image.id.

@norrietaylor @nick-alayil any preferences on how we should proceed with this ECS' adjustment?

[norrietaylor]

@ChrsMark, sorry for the late reply. This slipped through the cracks.

The first step will be to ensure the change is made in the ECS repository. Once that is complete, we can make code changes to the integrations. The plan is to make the Kibana dashboard backward-compatible for both variants of the data model.

@ChrsMark, will you be submitting a change to ECS?

We will need help from @kfirpeled and his team to make the change in Kibana.

I will find resources from my team to make the agent changes for both cloud-defend and Endpoint.

[ChrsMark]

Hey @norrietaylor ! I have filed #2282 with the changes.