Tweak logging style to have everything on console and on disk in JSON

[thbkrkr]

On ECK, we investigate a simple way that allows users to enable Stack monitoring in restricted Kubernetes environments. This means extracting log and stack monitoring metrics without the need for elevated permission or the ability to deploy DaemonSets in a Kubernetes environment. In this context, we want to run filebeat in a sidecar container configured to ship Elasticsearch log files. Related issue: elastic/cloud-on-k8s#4183.

There is a convenient way to enable disk logging with the recent addition of #62758 by setting ES_LOG_STYLE=file but this comes with the "drawback" of changing which logs and with what layout there are written to the console, compared to what we have with ES_LOG_STYLE=docker.

Today, by default, ECK users are used to having all logs on the console in JSON (current behaviour provided by ES_LOG_STYLE=docker). We don't want to break this.

In short, we would like to keep all logs written to the console in JSON and also log everything to disk.

We could achieve that by providing our custom log4j configuration but ideally we would prefer that ECK not handle this. Especially because there may be differences depending on the versions of Elasticsearch.

Do you think that another style of logging could be possible, or that the style of logging could be more easily configurable?

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

we discussed this on 12may es-delivery sync.
I think this is a good idea. We could support ES_LOG_STYLE=file to also log to a file? Would this be fine by cloud @elastic/cloud-delivery ?
Otherwise we could add a third option both/tee
Or we could allow to set ES_LOG_STYLE with both values - not sure how complex it would be, but some work would be required.

cc @elastic/es-delivery

[thbkrkr]

@pgomulka Thank you for looking into that!

We could support ES_LOG_STYLE=file to also log to a file?

Did you mean ES_LOG_STYLE=docker because ES_LOG_STYLE=file already logs to a file?

Would this be fine by cloud @elastic/cloud-delivery?

I don't know how interested @elastic/cloud-delivery is in this enhancement. The ECK team has a separate alias: @elastic/cloud-k8s. 😃