-
Notifications
You must be signed in to change notification settings - Fork 0
Interpretation of Exceptions
Ryan Slominski edited this page Dec 4, 2018
·
8 revisions
Many of the exceptions you see are a result of a lack of available resources (CPU, Memory, Disk, Network). This is especially true of the timeout related exceptions. Also some exceptions may be due to race conditions that are more likely to occur during high load / resource contention. Keep an eye on the resource monitor of the test machine.
During the "slow" test case the goal is to simulate slow clients in which the server is writing faster than the client is reading. This results in the following log message on the CAServer:
WARNING: Failed to send message to /127.0.0.1:54007 - buffer full.
This seems like a reasonable way to handle this scenario.
Not exactly sure what this one means. It appears to occur more frequently the higher the load on the server. Maybe a race condition of some kind exists internally in the EPICS JCA/CAJ library? Appears a write operation is attempted on a socket that is already closed.
WARNING: Exception caught when responding to event add request for channel with SID : 347
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at com.cosylab.epics.caj.cas.CASTransport.send(CASTransport.java:467)
at com.cosylab.epics.caj.cas.CASTransport.submit(CASTransport.java:653)
at com.cosylab.epics.caj.impl.requests.AbstractCARequest.submit(AbstractCARequest.java:88)
at com.cosylab.epics.caj.cas.handlers.EventAddResponse.eventResponse(EventAddResponse.java:174)
at com.cosylab.epics.caj.cas.CASServerMonitor.run(CASServerMonitor.java:137)
at com.cosylab.epics.caj.cas.CASTransport.run(CASTransport.java:953)
at java.lang.Thread.run(Thread.java:748)
Timeout while connecting to server.
SEVERE: null
gov.aps.jca.CAException: Failed to initialize reactor.
at com.cosylab.epics.caj.CAJContext.internalInitialize(CAJContext.java:748)
at com.cosylab.epics.caj.CAJContext.initialize(CAJContext.java:687)
at org.jlab.jts.caclient.caj.CAJClient.<init>(CAJClient.java:96)
at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.jlab.jts.integration.TestCaseRunner.run(TestCaseRunner.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at java.nio.channels.Selector.open(Selector.java:227)
at com.cosylab.epics.caj.impl.reactor.Reactor.initialize(Reactor.java:245)
at com.cosylab.epics.caj.impl.reactor.Reactor.<init>(Reactor.java:235)
at com.cosylab.epics.caj.CAJContext.internalInitialize(CAJContext.java:709)
... 10 more
Caused by: java.net.ConnectException: Connection timed out: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:127)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
... 21 more
Server connection already established, but timeout while connecting to specific EPICS channels.
SEVERE: Test Case Exception:
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at org.jlab.jts.integration.TestCaseRunner.run(TestCaseRunner.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at org.jlab.jts.caclient.caj.CAJChannelGroup.lambda$connectAsync$0(CAJChannelGroup.java:120)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Twenty seconds elapsed and a lock still could not be obtained. Might be high contention / heavy load. Might be actual deadlock.
java.io.IOException: Unable to close channel
at org.jlab.epics2web.epics.ChannelMonitor.close(ChannelMonitor.java:126)
at org.jlab.epics2web.epics.ChannelManager.removeFromChannels(ChannelManager.java:333)
at org.jlab.epics2web.epics.ChannelManager.removeListener(ChannelManager.java:353)
at org.jlab.epics2web.websocket.WebSocketSessionManager.removeClient(WebSocketSessionManager.java:169)
at org.jlab.epics2web.Application$1.run(Application.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: gov.aps.jca.CAException: Failed to obtain synchronization lock for 'R011PROT.F', possible deadlock.
at com.cosylab.epics.caj.CAJContext.destroyChannel(CAJContext.java:1017)
at com.cosylab.epics.caj.CAJChannel.destroy(CAJChannel.java:379)
at com.cosylab.epics.caj.CAJChannel.destroy(CAJChannel.java:366)
at org.jlab.epics2web.epics.ChannelMonitor.close(ChannelMonitor.java:124)
... 9 more