Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTEL: Exceptions not always shown human-readable in Kibana because of parsing result #401

Open
oldium opened this issue Nov 25, 2024 · 0 comments

Comments

@oldium
Copy link

oldium commented Nov 25, 2024

Hi, this might be simply parsing issue or Kibana display issue, I do not know. We are using OpenTelemetry to send logs and traces via APM Server into Elastic, our version is 8.16.0.

if err := setExceptionStacktrace(exceptionStacktrace, language, exceptionError.Exception); err != nil {
// Couldn't parse stacktrace, set it as `error.stack_trace` instead.
exceptionError.Exception.Stacktrace = nil
exceptionError.Exception.Cause = nil
exceptionError.StackTrace = exceptionStacktrace
}

This code either fills Error.Exception.Stacktrace (object with various attributes) or Error.StackTrace (string). Kibana is unfortunately not able to display usable (human-readable) stack traces in Logs Explorer if they come parsed (see screenshots for comparison), according to code the Kibana has some code able to render parsed exceptions, but I have not seen it working neither in APM Errors, Traces nor in Logs Explorer.

For the unparsed message there would have to be a more tolerant parser, because the stack trace looks like the following:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 501 Not Implemented
POST https://analytics.googleapis.com/upload/analytics/v3/management/accounts/54242820/webproperties/UA-54242820-11/customDataSources/De9KaW0jR8ufoMGGZv0epQ/uploads?uploadType=resumable
{
  "code": 501,
  "errors": [
    {
      "domain": "global",
      "message": "Operation is not implemented, or supported, or enabled.",
      "reason": "notImplemented"
    }
  ],
  "message": "Operation is not implemented, or supported, or enabled.",
  "status": "UNIMPLEMENTED"
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:583)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
        <further lines>

Parsed error with no human-readable stack trace in Logs Explorer:
Parsed error with no human-readable stack trace

Unparsed error (there is a POST request logged as part of stack trace from Google library):
Unparsed error

Anyway, we see no exceptions in APM Errors, though.

No stack trace in APM Errors display:
No stack trace

The trace document contains parsed exception:
Trace document

Kibana is trying to detect exceptions even in the parsed state, but the check is actually useless and looks like a Kibana bug - the error.exception.stacktrace does not have any direct value for parsed exceptions, they are stored in parallel-structure sub-attributes error.exception.stacktrace.*:

https://github.com/elastic/kibana/blob/a41017e74e4f495c6892fbe023cff0045839bd9b/packages/kbn-discover-utils/src/utils/get_stack_trace_fields.ts#L18-L20

We used APM Java Agent until now and this always worked, so to me this looks more like an issue in conversion from OpenTelemetry messages into Elastic documents rather than Kibana display issue.

Is any more information needed? Should I create also Kibana issue? Feel free to suggest next steps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant