Ensure all data plane components in the path of requests output stack traces for 500 errors

[R-Lawton]

WHAT:

Ensure all data plane components in the path of requests (authorino, limitador, wasm-shim) output stack traces for 500 errors when there is a server error originating from them

[david-martin]

Authorino uses zap for structured logging. More details about how it's configured in https://docs.kuadrant.io/authorino/docs/user-guides/observability/#log-levels-and-log-modes

example log output

{"level":"info","ts":"2024-03-22T15:34:57Z","logger":"authorino","msg":"setting instance base logger","min level":"info","mode":"production"}
{"level":"info","ts":"2024-03-22T15:34:57Z","logger":"authorino","msg":"booting up authorino","version":"991b6ebc2b3b415d2e5b12f54c8fe8959b868212","cmd":"server"}
{"level":"info","ts":"2024-03-22T15:34:57Z","logger":"authorino","msg":"starting http auth service","port":5001,"tls":false}
{"level":"info","ts":"2024-03-22T15:34:57Z","logger":"authorino","msg":"starting grpc auth service","port":50051,"tls":false}

[david-martin]

Limitador uses https://docs.rs/tracing-log/latest/tracing_log/ ("Adapters for connecting unstructured log records from the log crate into the tracing ecosystem")
The log crate is a logging facade.
Log levels can be set in limitador via the RUST_LOG env var.

Logging is not structured at this time.

As limitador compiles to a binary, stacktraces will not be 'proper' as symbols are not included in production images.

Example log output from limitador when something fails:

2024-03-22T16:16:44.334295Z ERROR should_rate_limit: limitador_server::envoy_rls::server: Error: Storage("Reconnecting failed: Connection refused (os error 61)")  

[eguzki]

Limitador recently switched to tokio-tracing for logging in Kuadrant/limitador#250. It is a framework to collect structured info, so it should not be hard to structure log in any format you want.

[david-martin]

wasm-shim is loaded as a container image inside envoy, so any logs from the wasm-shim are output in the envoy logs.
In the case of istio, this would be the istio gateway pod that runs envoy e.g.

kubectl get po -A | grep -i istio 
...
kuadrant-system      api-gateway-istio-8748c5947-sxk8f

Example of log output in envoy

2024-03-26T12:33:04.818862Z	info	wasm	fetching image kuadrant/wasm-shim from registry quay.io with tag v0.3.0
2024-03-26T12:33:06.392767Z	info	Readiness succeeded in 1.695333064s
2024-03-26T12:33:06.392961Z	info	Envoy proxy is ready

log messages from wasm modules are proxied as strings, so you wouldn't get any structured logging in enovys log output

[david-martin]

I'm closing this out with the information given.
That can be taken into account with the related docs issue for a tracing guide Kuadrant/docs.kuadrant.io#70