[Bug] - ElasticSearch javassist.NotFoundException: java.lang.Object[]

[ChaosMarc]

Hi,

I'm trying to add the latest inspectIT Ocelot Agent to an ElasticSearch v7.17.1 installation (operator pattern). Everything is running in an Kubernetes Cluster.

After mounting the jar file to the container and adding "-javaagent:/usr/share/elasticsearch/data/libs/inspectit-ocelot-agent.jar" to the jvm options (via ES_JAVA_OPTS env) I first was presented with a lot of runtime permission exceptions which I could fix by adding a .java.policy file. Not sure if all those listed permissions are really needed, but at least there were no more exceptions regarding missing permissions...

Now I'm running into javassist.CannotCompileException: [source error] javassist.NotFoundException: java.lang.Object[] exceptions and can't find a way around them. The ElasticSearch container continues to start regardless of the exceptions but doesn't work properly afterwards (my connected liferay instance throws errors when trying to use the ES index)

I've attached the ElasticSearch debug logs which were created with the following relevant arguments:
-Xshare:off -Dinspectit.instrumentation.special.class-loader-delegation=false -Dinspectit.publishOpenCensusToBootstrap=false -Dinspectit.logging.debug=true -Dinspectit.self-monitoring.enabled=true
es.log

[aaronweissler]

Hi @ChaosMarc ,

the underlying issue seems to still be missing permissions, because the error does not occur anymore in my testing when simply granting all permissions.

Since granting all permissions to everything defeats the purpose of using a SecurityManager in the first place, I would recommend granting all permissions to only the agent by using the following in your policy:

grant codeBase "file:<absolute_path_to_inspectit-ocelot-agent.jar>" {
    permission java.security.AllPermission;
};

Let me know if this fixes your problems.

Best regards,
Aaron

[ChaosMarc]

Hi Aaron,

thank you very much for your support. With your java policy (which should have been the obvious solution to me...) there are no more exceptions and elasticsearch starts up fully functional BUT the agent is not reachable under the specified port (7777 in my case):

kubectl exec --namespace='liferay' elasticsearch-0 -c elasticsearch -it -- sh

sh-5.0# curl http://localhost:7777
curl: (7) Failed to connect to localhost port 7777: Connection refused

This is the full startup log: https://gist.github.com/ChaosMarc/3536bf7df89dfe4deeb1d04a6e02817b

Any ideas on what I'm doing wrong now? :D

Kind regards
Marc

[mariusoe]

Hi @ChaosMarc

since version 1.15.x the Prometheus exporter isn't enabled by default anymore. See this breaking change note.

If you want to use Prometheus, you have to explicitly enable it which can be done using the following flag:

-Dinspectit.exporters.metrics.prometheus=ENABLED

[ChaosMarc]

this is the kind of thing that happens when you think you can use the latest version of some software for testing instead of the one you actually have in production -.- thanks for pointing out the change. after adding -Dinspectit.exporters.metrics.prometheus .enabled=ENABLED to my JAVA_OPTS everything works fine. thank you very much again!