WebSocket DeadLock In Javalin 5.6.1 Jetty 11.0.15

[dlwss]

Jetty version(s)

• Javalin 5.6.1
• Jetty 11.0.15 (from maven central as i search)
• JDK azul-11.0.19

OS type/version
centos7

Description
I used Javalin's WebSocket server, and when I ran it for a while, I found that from time to time, I would not receive messages from
the client

How to reproduce?
After running for a while

Additional context

you can see ,A lock race occurred between thread 39 and thread 41。
Also，It is a Jetty Code

[arthas@2470]$ thread -b
"JettyServerThreadPool-39-acceptor-1@4ce9d56b-ServerConnector@6075b2d3{HTTP/1.1, (http/1.1)}{0.0.0.0:8887}" Id=39 RUNNABLE
    at java.base@11.0.11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at java.base@11.0.11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
    at java.base@11.0.11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
    at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
    at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)

    Number of locked synchronizers = 1
    - java.util.concurrent.locks.ReentrantLock$NonfairSync@223fa435 <---- but blocks 3 other threads!

[arthas@2470]$ thread 39
"JettyServerThreadPool-39-acceptor-1@4ce9d56b-ServerConnector@6075b2d3{HTTP/1.1, (http/1.1)}{0.0.0.0:8887}" Id=39 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@223fa435 owned by "JettyServerThreadPool-41-acceptor-3@27d95d1f-ServerConnector@6075b2d3{HTTP/1.1, (http/1.1)}{0.0.0.0:8887}" Id=41
    at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@223fa435
    at java.base@11.0.11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
    at java.base@11.0.11/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
    at java.base@11.0.11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:275)
    at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
    at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)

[arthas@2470]$ thread 41
"JettyServerThreadPool-41-acceptor-3@27d95d1f-ServerConnector@6075b2d3{HTTP/1.1, (http/1.1)}{0.0.0.0:8887}" Id=41 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@223fa435 owned by "JettyServerThreadPool-39-acceptor-1@4ce9d56b-ServerConnector@6075b2d3{HTTP/1.1, (http/1.1)}{0.0.0.0:8887}" Id=39
    at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@223fa435
    at java.base@11.0.11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
    at java.base@11.0.11/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
    at java.base@11.0.11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:275)
    at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
    at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)

then ,I looked at the Jetty and JDK code

Jetty

    @Override
    public void accept(int acceptorID) throws IOException
    {
        ServerSocketChannel serverChannel = _acceptChannel;
        if (serverChannel != null && serverChannel.isOpen())
        {
            SocketChannel channel = serverChannel.accept();
            accepted(channel);
        }
    }

JDK11

@Override
    public SocketChannel accept() throws IOException {
        acceptLock.lock();
        try {
            int n = 0;
            FileDescriptor newfd = new FileDescriptor();
            InetSocketAddress[] isaa = new InetSocketAddress[1];

            boolean blocking = isBlocking();
            try {
                begin(blocking);
                do {
                    n = accept(this.fd, newfd, isaa);
                } while (n == IOStatus.INTERRUPTED && isOpen());
            } finally {
                end(blocking, n > 0);
                assert IOStatus.check(n);
            }

            if (n < 1)
                return null;

            // newly accepted socket is initially in blocking mode
            IOUtil.configureBlocking(newfd, true);

            InetSocketAddress isa = isaa[0];
            SocketChannel sc = new SocketChannelImpl(provider(), newfd, isa);

            // check permitted to accept connections from the remote address
            SecurityManager sm = System.getSecurityManager();
            if (sm != null) {
                try {
                    sm.checkAccept(isa.getAddress().getHostAddress(), isa.getPort());
                } catch (SecurityException x) {
                    sc.close();
                    throw x;
                }
            }
            return sc;

        } finally {
            acceptLock.unlock();
        }
    }

I think it's the problem of the Jetty caller, but I can't find the root cause。

I also report this issue in javalin

[sbordet]

This is completely normal and it's not a deadlock.

Jetty uses multiple threads for accepting as explained here:
https://eclipse.dev/jetty/documentation/jetty-12/programming-guide/index.html#pg-server-http-connector-acceptors

Please also consider closing the Javalin issue, as it is completely normal behavior.

@gregw time to set the default number of acceptors to 1?

[dlwss]

Thank you very much for your reply.
I looked at the link address of the document, there is a doubt want to consult.
Now the problem is that when a new link comes in, the lock competition that appears above will occasionally appear (it may be a long time, a few seconds or ten minutes). The explanation given in the documentation is

 "When a TCP connection is accepted, ServerConnector wraps the accepted SocketChannel and passes it to the SelectorManager. Therefore,  there is a little moment where the acceptor thread is not accepting new connections because it is busy wrapping the just  accepted connection to pass it to the SelectorManager."

If I set acceptQueueSize to 1, does this still happen when I have multiple WebSocket links?

Or how do I set up to avoid new WebSocket links that do not respond to messages due to acceptors?

[gregw]

@dlwss So the idea with the lock and multiple acceptor threads is that at any time only 1 thread is do an accept call, but if you have N acceptor threads, then N-1 should be waiting at that lock to start accepting as soon as a connection has been accepted by the first thread and it is processing it.

In practise, the time it takes to create a new connection around a newly accepted thread is very very small, so the "little moment" is typically not a concern. Thus acceptors==1 is a reasonable value and we should make it the default..... actually I think we also support acceptors==0, for asynchronous accepting of connections from the selectors. @sbordet would that be a better default?

[sbordet]

@dlwss acceptQueueSize is a different setting and has little to do with the number of acceptors.
You seem to be worrying about the wrong details.

Or how do I set up to avoid new WebSocket links that do not respond to messages due to acceptors?

This is not clear. What's exactly a "WebSocket link"?

Can you please state exactly your problem, not the solution you may have found, or Jetty implementation details (or configuration parameters) that you think are a problem, but they are not?

Once you state your problem in detail, we can understand better what's going on and provide more detailed answers/solution to your problem.