Query failed because of the timer return negative value

[uniray7]

Hi, I am using the trino 363.
And recently, the error "EXTERNAL ERROR — GENERIC_INTERNAL_ERROR" happened frequently.

java.lang.IllegalArgumentException: value is negative
	at io.airlift.units.Preconditions.checkArgument(Preconditions.java:26)
	at io.airlift.units.Duration.<init>(Duration.java:69)
	at io.airlift.units.Duration.succinctDuration(Duration.java:59)
	at io.airlift.units.Duration.succinctNanos(Duration.java:54)
	at io.airlift.units.Duration.nanosSince(Duration.java:49)
	at io.trino.server.remotetask.TaskInfoFetcher.updateStats(TaskInfoFetcher.java:283)
	at io.trino.server.remotetask.TaskInfoFetcher.success(TaskInfoFetcher.java:245)
	at io.trino.server.remotetask.TaskInfoFetcher.success(TaskInfoFetcher.java:51)
	at io.trino.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:49)
	at io.trino.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:27)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Those queries are just start executing but then failed

I think it might be related to the code snippet:
stats.infoRoundTripMillis(nanosSince(currentRequestStartNanos).toMillis());
ref
And the nanosSince(currentRequestStartNanos) will check if it is negative
ref

[raunaqmorarka]

I've also started observing these failures recently on our internal CI jobs.
The current usages of Duration.nanosSince are based on the expectation that System#nanoTime will always return monotonically increasing values. It seems that somehow that assumption is being violated sporadically.
Is your observation of these failures correlated with any recent change to the OS or JVM version used in your environment ?

[uniray7]

Actually, we deployed on k8s pod.
The pods was started from the end of January and we didn't do any deployment recently, so the JVM OS are not changed.
And these error begin happening since around 2/12.