JMX monitoring via "v1/jmx/mbean" endpoint is not discussed in trino documentation

[mgorbatenko]

Hey all! I was in the process of setting up jolokia for trino 411, specifically for the REST interface to JMX, when I came across this comment from 2021. The v1/jmx/mbean endpoint seems like a durable, useful, and working solution for JMX monitoring, but I couldn't find any mention of this in the official docs. It's readily available via HTTP and essentially any measure can be parsed.

Is there a reason this isn't in the docs? I would hate to start using it and then have the feature disappear or become unsupported. If there isn't a reason, I could take a stab at updating the docs to include a reference to this endpoint.

[raunaqmorarka]

fyi @mosabua

[hashhar]

@dain / @electrum on why it exists and whether it's something that's ok to use externally.

[mosabua]

From what I know you can just use a JMX connection directly via dedicated ports as described in
https://trino.io/docs/current/admin/jmx.html

You don't need to go through the API. Also the JMX connector is available for SQL based access to the info. https://trino.io/docs/current/connector/jmx.html

If we want to document the REST endpoint for the mbean access it should be documented in
https://trino.io/docs/current/admin/jmx.html

Also related to some degree.. and also not yet documented .. we have opentelemtry support now as well..

[mgorbatenko]

This is helpful! Thank you @mosabua.

[ToanP]

Can someone help to resolve jmx does not work?

$ cat jvm.config
-server
-Xmx8G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-Djdk.attach.allowAttachSelf=true
-XX:-UseBiasedLocking
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.nio.maxCachedBufferSize=2000000
-XX:+UnlockDiagnosticVMOptions
-XX:+UseAESCTRIntrinsics
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9080
-Dcom.sun.management.jmxremote.rmi.port=9081
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

$ cat config.properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
query.max-memory=3TB
query.max-memory-per-node=5GB
memory.heap-headroom-per-node=2GB
discovery.uri=http://localhost:8080
spill-enabled=false
spiller-spill-path="/cache"
max-spill-per-node=300GB
query-max-spill-per-node=300GB
spill-compression-enabled=false
retry-policy=TASK
query.client.timeout=30s
jmx.rmiregistry.port=9080
jmx.rmiserver.port=9081

I configured jmx as above and opened port-forward 9080:9080. But JConsole, VisualVM can't get any jmx informtion. jConsole shows "Connecting" forever and VisualVM shows nothing.

The connection is: localhost:9080

[mosabua]

We updated JMX docs a while ago .. do you have the ports open as documented?

Specifically do you have -Dcom.sun.management.jmxremote.rmi.port=9081 in the jvm config?

I am not sure if we consider the /v1/jmx/mbean part of the REST API ... I kinda think not .. either go directly via JMX or use the JMX connector. Going to JMX info via REST seems convoluted.

[mosabua]

I think we can close this ticket .. if you cant get JMX directly to work please reopen with more details.