Ocelot causes java.lang.NumberFormatException in Confluence

[Marty]

I'm attaching inspectit-ocelot-agent-1.6.1.jar without any configuration to my confluence 7.4.1 instance. This breaks some functionality in a self-developed plugin, causing the following exception:

org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
        java.lang.NumberFormatException: 55e8d2ee576bc05019b44411ed56c8e7 should be a <=16 character lower-hex string with no prefix
                at com.github.kristofa.brave.IdConversion.convertToLong(IdConversion.java:41)
                at com.github.kristofa.brave.http.HttpServerRequestAdapter.getSpanId(HttpServerRequestAdapter.java:54)
                at com.github.kristofa.brave.http.HttpServerRequestAdapter.getTraceData(HttpServerRequestAdapter.java:31)
                at com.github.kristofa.brave.ServerRequestInterceptor.handle(ServerRequestInterceptor.java:33)
                at com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:56)
                at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)

When I disable ocelot everything works again and the exception does no longer occur.
Can you provide some idea as to why this happens and how I can get around that?

[MariusBrill]

Hi @Marty, i suspect that this may be an error caused by the agent writing into http-headers in requests.

For clarification: The agent ships with a default config. This config is designed to provide some standard metrics out of the box. Part of these are functions which add http-headers. These are used to recreate traces between services.

To verfiy this, we need to deactivate the scopes that create these headers. In order to do so, please add a file with the following content to your config:

inspectit:
  instrumentation:
    rules:
    
      'r_apacheclient_servicegraph_propagation':
        scopes:
          's_apacheclient_doExecute': false
          
      'r_apacheclient_servicegraph_record':
        scopes:
          's_apacheclient_doExecute': false
      
      'r_apacheclient_tracing':
        scopes:
          's_apacheclient_doExecute': false
          
      'r_apacheclient_propagation':
        scopes:
          's_apacheclient_doExecute': false
          
      'r_httpurlconnection_http_metric':
        scopes:
          's_httpurlconnection_getInputStream': false
          's_httpurlconnection_requestInitiators': false
          
      'r_httpurlconnection_down_propagation':
        scopes:
          's_httpurlconnection_connect': false
          's_httpurlconnection_getOutputStream': false
          's_httpurlconnection_getInputStream': false
          
      'r_httpurlconnection_up_propagation':
        scopes:
          's_httpurlconnection_getInputStream': false
          
      'r_httpurlconnection_servicegraph_propagation':
        scopes:
          's_httpurlconnection_connect': false
          's_httpurlconnection_getOutputStream': false
          's_httpurlconnection_getInputStream': false
      
      'r_httpurlconnection_servicegraph_record':
        scopes:
          's_httpurlconnection_getInputStream': false
          's_httpurlconnection_requestInitiators': false
          
      'r_httpurlconnection_tracing_start':
        scopes:
          's_httpurlconnection_connect': false
          's_httpurlconnection_getOutputStream': false
          
      'r_httpurlconnection_tracing_end':
        scopes:
          's_httpurlconnection_getInputStream': false
          's_httpurlconnection_requestInitiators': false
          
      'r_servletapi_http_metric':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false
          
      'r_servletapi_downPropagation':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false
          
      'r_servletapi_servlet_filter_upPropagation':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false
          
      'r_servletapi_servletresponse_upPropagation':
        scopes:
          's_servletapi_servletresponse_getWriter': false
          's_servletapi_servletresponse_getOutputStream': false
          
      'r_servletapi_servicegraph_propagation':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false
          's_servletapi_servletresponse_getWriter': false
          's_servletapi_servletresponse_getOutputStream': false
          
      'r_servletapi_servicegraph_record':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false
          
      'r_servletapi_tracing':
        scopes:
          's_servletapi_servlet_service': false
          's_servletapi_filter_doFilter': false

and check if the error still occurs.

[Marty]

Yes this helped, thank you very much!

Can you elaborate on what this does or point me to the docs for this?

[MariusBrill]

Unfortunately there is no documentation for the default configuration yet, however you can lookup the config here.

What i send you basically turns off all rules connected to the http headers by deactivating all scopes they use. You can read more about that in our documentation.
This however also limits the data you gain from the agent. You can try to re-enable these rules one at a time to find out which rule exactly causes the troubles.

As for the http headers themselves: We use them to trace back requests between services. If you are interested you can check out one of our demos . We use the data from these headers to build our Service Graph you can find there.

[MariusBrill]

Hi @Marty,
i hope you were able to make the agent work as you expected!

Since we are currently working on monitoring solutions for confluence and jira (and especially improving the user experience), could you tell us what your use cases for the inspectIT Agent are and what you plan to use the agent for?

Cheers
Marius

[Marty]

@MariusBrill Sort of. The problem of this issue is solved, now I ran into the next one (will open issue in a moment).

We want to monitor the health (gc, ram, cpu ..) of our confluence/jira instances and notice early when something goes sideways to prevent any crashes/downtimes of the systems.
A few plugins cause all sorts of problem with open file descriptors or memory issues.