Jetty问题分析

Jetty 9.4.17版本bug分析

问题描述

使用了9.4.17版本的jetty作为容器，当服务启动后，过段时间发起http请求就会没有响应，通过抓包发现客户端和服务端连接都已建立。通过分析工具查看，发现jetty的acceptor已经接受了请求，但是没有线程进行处理。

正常情况:

异常情况:

原因分析

本人外网电脑CPU是i9-9900k正好16个cpu核心。通过搜资料查看Jetty的线程模型并下载源码，发现Jetty的一个请求调用涉及到acceptor，selector，reserved ，idle这四类线程，据我的理解idle线程即worker。acceptor负责监听servlet请求，selector负责请求解析转发，新的idle线程负责执行调用，reserved 线程池是jetty9.4.x的新功能，当jetty线程池不够的时候才会使用, 本问题不涉及。

通过查看源码，Jetty的servel首先一个很重要的配置文件即jetty-threadpool.xml

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">

<Configure>
  <!-- =========================================================== -->
  <!-- Configure the Server Thread Pool.                           -->
  <!-- The server holds a common thread pool which is used by      -->
  <!-- default as the executor used by all connectors and servlet  -->
  <!-- dispatches.                                                 -->
  <!--                                                             -->
  <!-- Configuring a fixed thread pool is vital to controlling the -->
  <!-- maximal memory footprint of the server and is a key tuning  -->
  <!-- parameter for tuning.  In an application that rarely blocks -->
  <!-- then maximal threads may be close to the number of 5*CPUs.  -->
  <!-- In an application that frequently blocks, then maximal      -->
  <!-- threads should be set as high as possible given the memory  -->
  <!-- available.                                                  -->
  <!--                                                             -->
  <!-- Consult the javadoc of o.e.j.util.thread.QueuedThreadPool   -->
  <!-- for all configuration that may be set here.                 -->
  <!-- =========================================================== -->
  <New id="threadPool" class="org.eclipse.jetty.util.thread.QueuedThreadPool">
    <Set name="minThreads" type="int"><Property name="jetty.threadPool.minThreads" deprecated="threads.min" default="10"/></Set>
    <Set name="maxThreads" type="int"><Property name="jetty.threadPool.maxThreads" deprecated="threads.max" default="200"/></Set>
    <Set name="reservedThreads" type="int"><Property name="jetty.threadPool.reservedThreads" default="-1"/></Set>
    <Set name="idleTimeout" type="int"><Property name="jetty.threadPool.idleTimeout" deprecated="threads.timeout" default="60000"/></Set>
    <Set name="detailedDump" type="boolean"><Property name="jetty.threadPool.detailedDump" default="false"/></Set>
  </New>
</Configure>

配合源码文件QueuedThreadPool可以看到

启动时会生成${minThreads}个qtp线程，但是实际上看代码会再启动n个idle线程。如下图所示：图中启动了3个，这个idleThread的数量由jetty启动时_job任务队列的长度以及进入循环的次数决定。

Jetty QueuedThreadPool初始化时生成idleThread的代码

private Runnable _runnable = new Runnable()
{
    @Override
    public void run()
    {
        boolean idle = false;

        try
        {
            Runnable job = _jobs.poll();
            if (job != null && _threadsIdle.get() == 0)
                startThreads(1);

            while (true)
            {
                if (job == null)
                {
                    if (!idle)
                    {
                        idle = true;
                        _threadsIdle.incrementAndGet();
                    }

                    if (_idleTimeout <= 0)
                        job = _jobs.take();
                    else
                    {
                        // maybe we should shrink?
                        int size = _threadsStarted.get();
                        if (size > _minThreads)
                        {
                            long last = _lastShrink.get();
                            long now = System.nanoTime();
                            if (last == 0 || (now - last) > TimeUnit.MILLISECONDS.toNanos(_idleTimeout))
                            {
                                if (_lastShrink.compareAndSet(last, now))
                                    break;
                            }
                        }

                        job = _jobs.poll(_idleTimeout, TimeUnit.MILLISECONDS);
                    }
                }

                // run job
                if (job != null)
                {
                    if (idle)
                    {
                        idle = false;
                        if (_threadsIdle.decrementAndGet() == 0)
                            startThreads(1);
                    }

                    if (LOG.isDebugEnabled())
                        LOG.debug("run {}", job);
                    runJob(job);
                    if (LOG.isDebugEnabled())
                        LOG.debug("ran {}", job);

                    // Clear interrupted status
                    Thread.interrupted();
                }

                if (!isRunning())
                    break;

                job = _jobs.poll();
            }
        }
        catch (InterruptedException e)
        {
            LOG.ignore(e);
        }
        catch (Throwable e)
        {
            LOG.warn(String.format("Unexpected thread death: %s in %s", this, QueuedThreadPool.this), e);
        }
        finally
        {
            if (idle)
                _threadsIdle.decrementAndGet();

            removeThread(Thread.currentThread());

            if (_threadsStarted.decrementAndGet() < getMinThreads())
                startThreads(1);
        }
    }
};

P.S. 我们在刚启动的时候能够调用成功正是因为这三个进程没有退出，根据idleTimeout的设置，当60s*n的空闲时间时间过去后，这3个线程也依次被销毁，没有空闲的线程供外部请求使用了，jetty容器对于我们而言也就没有响应了，所以增加idleTimeout也是可以续命的 :)，不过这是后话了。

同时看代码AbstractConnector可以看到，16核心的CPU可以会启动2个acceptor (8个核心则只有1个acceptor)

看代码SelectorManager可知，16核心的CPU可以会启动8个selector(8个核心则只有4个selector)

所以理论上，如果没有空闲线程(10 - 2 - 8 = 0)，jetty一定会夯死。github上开发人员的修复方案是保证时刻有一个idleThread。