Using HttpClient properly to avoid CLOSE_WAIT TCP connections

简介:          Apache的HttpComponent(这里是基于 version 4.1)组件,用的人不在少数。但是能用好的人,却微乎其微,为什么?很简单,TCP/IP协议里面的细节太多了(细节是魔鬼),像并发请求控制&资源释放,Nagle算法参数优化,Connection eviction,跟ulimit配对的total connection,重定向策略定制化,两类超时时间

         Apache的HttpComponent(这里是基于 version 4.1)组件,用的人不在少数。但是能用好的人,却微乎其微,为什么?很简单,TCP/IP协议里面的细节太多了(细节是魔鬼),像并发请求控制&资源释放,Nagle算法参数优化,Connection eviction,跟ulimit配对的total connection,重定向策略定制化,两类超时时间的合理设置,流读写等等。

         在最近的项目中,更是破天荒的遇到了close_wait问题,所以利用业余时间索性将之前同学写的HttpClient优化了一遍。下面我将贴出代码,如果大家发现了还有改进的余地,记得千万要留言知会我,共创最棒的代码:

/**
 * 史上最棒的HttpClient4封装,details please see
 * http://hc.apache.org/httpcomponents-client-ga/tutorial/html/index.html
 * 
 * @author von gosling 2013-5-7
 */
public class HttpClientManager {

    //Consider ulimit
    private static final int                   DEFAULT_MAX_TOTAL_CONNECTIONS     = 7500;
    //notice IE 6,7,8  
    private static final int                   DEFAULT_MAX_CONNECTIONS_PER_ROUTE = 200;

    private static final int                   DEFAULT_CONN_TIMEOUT_MILLISECONDS = 5 * 1000;

    private static final int                   DEFAULT_READ_TIMEOUT_MILLISECONDS = 60 * 1000;

    private static final int                   INIT_DELAY                        = 5 * 1000;

    private static final int                   CHECK_INTERVAL                    = 5 * 60 * 1000;

    private static String                      HTTP_REQUEST_ENCODING             = "UTF-8";
    private static String                      LINE_SEPARATOR                    = "\r\n";

    private static final Logger                LOG                               = LoggerFactory
                                                                                         .getLogger(HttpClientManager.class);

    private static ThreadSafeClientConnManager connectionManager;
    static {
        SchemeRegistry schemeRegistry = new SchemeRegistry();
        schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
        //schemeRegistry.register(new Scheme("https", 443, SSLSocketFactory.getSocketFactory()));

        connectionManager = new ThreadSafeClientConnManager(schemeRegistry);
        connectionManager.setMaxTotal(DEFAULT_MAX_TOTAL_CONNECTIONS);
        connectionManager.setDefaultMaxPerRoute(DEFAULT_MAX_CONNECTIONS_PER_ROUTE);

        //Connection eviction
        ScheduledExecutorService scheduledExeService = Executors.newScheduledThreadPool(1,
                new DaemonThreadFactory("Http-client-ConenctionPool-Monitor"));
        scheduledExeService.scheduleAtFixedRate(new IdleConnectionMonitor(connectionManager),
                INIT_DELAY, CHECK_INTERVAL, TimeUnit.MILLISECONDS);
    }

    public static String doPost(String reqURL, Map<String, String> params, String encoding,
                                Boolean enableSSL) {
        HttpClient httpClient = getHttpClient(enableSSL);

        String responseContent = "";
        try {
            HttpPost httpPost = buildHttpPostRequest(reqURL, params, encoding);
            HttpResponse response = httpClient.execute(httpPost);

            //            validateResponse(response, httpPost);

            HttpEntity entity = response.getEntity();
            if (entity != null) {
                // responseLength = entity.getContentLength();
                responseContent = EntityUtils.toString(entity, encoding);
                //Ensure that the entity content has been fully consumed and the underlying stream has been closed.
                EntityUtils.consume(entity);
            } else {
                LOG.warn("Http entity is null! request url is {},response status is {}", reqURL,
                        response.getStatusLine());
            }
        } catch (ConnectTimeoutException e) {
            LOG.warn(e.getMessage());
        } catch (SocketTimeoutException e) {
            LOG.warn("Read time out!");
        } catch (SSLPeerUnverifiedException e) {
            LOG.warn("Peer not authenticated!");
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }
        return responseContent;
    }

    public static String doPost(String reqURL, final String entities, String encoding) {
        HttpClient httpClient = getHttpClient(false);

        String responseContent = "";
        try {
            AbstractHttpEntity printWriterEntity = new AbstractHttpEntity() {
                public boolean isRepeatable() {
                    return false;
                }

                public long getContentLength() {
                    return -1;
                }

                public boolean isStreaming() {
                    return false;
                }

                public InputStream getContent() throws IOException {
                    // Should be implemented as well but is irrelevant for this case
                    throw new UnsupportedOperationException();
                }

                public void writeTo(final OutputStream outstream) throws IOException {
                    PrintWriter writer = new PrintWriter(new OutputStreamWriter(outstream,
                            HTTP_REQUEST_ENCODING));
                    writer.print(entities);
                    writer.print(LINE_SEPARATOR);
                    writer.flush();
                }

            };
            HttpPost httpPost = new HttpPost(reqURL);
            //If the data is large enough that you need to stream it,
            //you can write to a temp file and use FileEntity or possibly set up a pipe and use InputStreamEntity
            httpPost.setEntity(printWriterEntity);
            HttpResponse response = httpClient.execute(httpPost);

            validateResponse(response, httpPost);

            HttpEntity entity = response.getEntity();
            if (entity != null) {
                responseContent = EntityUtils.toString(entity, encoding);
                //Ensure that the entity content has been fully consumed and the underlying stream has been closed.
                EntityUtils.consume(entity);
            } else {
                LOG.warn("Http entity is null! request url is {},response status is {}", reqURL,
                        response.getStatusLine());
            }
        } catch (SocketTimeoutException e) {
            LOG.warn("Read time out!");
        } catch (SSLPeerUnverifiedException e) {
            LOG.warn("Peer not authenticated!");
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }
        return responseContent;
    }

    private static X509TrustManager customTrustManager(HttpClient httpClient) {
        //Trusting all certificates
        X509TrustManager xtm = new X509TrustManager() {
            public void checkClientTrusted(X509Certificate[] chain, String authType)
                    throws CertificateException {
            }

            public void checkServerTrusted(X509Certificate[] chain, String authType)
                    throws CertificateException {
            }

            public X509Certificate[] getAcceptedIssuers() {
                return null;
            }
        };
        try {
            SSLContext ctx = SSLContext.getInstance("TLS");
            if (null != ctx) {
                ctx.init(null, new TrustManager[] { xtm }, null);
                SSLSocketFactory socketFactory = new SSLSocketFactory(ctx,
                        SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
                httpClient.getConnectionManager().getSchemeRegistry()
                        .register(new Scheme("https", 443, socketFactory));
            }
        } catch (Exception e) {
            LOG.error(e.getMessage());
        }

        return xtm;
    }

    private static HttpClient getHttpClient(Boolean enableSSL) {
        DefaultHttpClient httpClient = new DefaultHttpClient(connectionManager);
        httpClient.setRedirectStrategy(new RedirectStrategy() { //设置重定向处理方式为自行处理
                    @Override
                    public boolean isRedirected(HttpRequest request, HttpResponse response,
                                                HttpContext context) throws ProtocolException {
                        return false;
                    }

                    @Override
                    public HttpUriRequest getRedirect(HttpRequest request, HttpResponse response,
                                                      HttpContext context) throws ProtocolException {
                        return null;
                    }
                });

        httpClient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT,
                DEFAULT_READ_TIMEOUT_MILLISECONDS);
        httpClient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
                DEFAULT_CONN_TIMEOUT_MILLISECONDS);
        //According to http use-case to decide to whether to open TCP_NODELAY option,So does SO_LINGER option 
        httpClient.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, Boolean.TRUE);
        httpClient.getParams().setParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,
                Boolean.FALSE);

        if (enableSSL) {
            customTrustManager(httpClient);
        }

        return httpClient;
    }

    public static Map.Entry<Integer, String> doGetHttpResponse(String url, String encoding) {
        HttpClient httpClient = getHttpClient(false);
        HttpGet httpget = new HttpGet(url);
        try {
            EncodingResponseHandler responseHandler = new EncodingResponseHandler();

            if (StringUtils.isBlank(encoding)) {
                encoding = HTTP_REQUEST_ENCODING;
            }
            responseHandler.setEncoding(encoding);

            return httpClient.execute(httpget, responseHandler);
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }
        return null;
    }

    public static String doGet(String url, String encoding) {
        Map.Entry<Integer, String> ret = doGetHttpResponse(url, encoding);
        if (ret == null) {
            return "";
        }
        if (ret.getKey() != HttpStatus.SC_OK) {
            LOG.error(
                    "Did not receive successful HTTP response: status code = {}, request url = {}",
                    ret.getKey(), url);
        }

        return ret.getValue();
    }

    public static void doPost(String url, Map<String, String> params) {
        HttpClient httpClient = getHttpClient(false);
        try {
            HttpPost httpPost = buildHttpPostRequest(url, params, HTTP.UTF_8);
            ResponseHandler<byte[]> handler = new ResponseHandler<byte[]>() {
                public byte[] handleResponse(HttpResponse response) throws ClientProtocolException,
                        IOException {
                    HttpEntity entity = response.getEntity();
                    if (entity != null) {
                        return EntityUtils.toByteArray(entity);
                    } else {
                        return null;
                    }
                }
            };
            httpClient.execute(httpPost, handler);
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }
    }

    private static HttpPost buildHttpPostRequest(String url, Map<String, String> params,
                                                 String encoding)
            throws UnsupportedEncodingException {
        HttpPost httpPost = new HttpPost(url);
        //Encode the form parameters
        if (!CollectionUtils.isEmpty(params)) {
            List<NameValuePair> nvps = Lists.newArrayList();
            Set<Entry<String, String>> paramEntrys = params.entrySet();
            for (Entry<String, String> entry : paramEntrys) {
                nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));
            }
            httpPost.setEntity(new UrlEncodedFormEntity(nvps, encoding));
        }
        return httpPost;
    }

    //    private static void validateResponse(HttpResponse response, HttpGet get) throws IOException {
    //        StatusLine status = response.getStatusLine();
    //        if (status.getStatusCode() >= HttpStatus.SC_MULTIPLE_CHOICES) {
    //            LOG.warn(
    //                    "Did not receive successful HTTP response: status code = {}, status message = {}",
    //                    status.getStatusCode(), status.getReasonPhrase());
    //            get.abort();
    //            return;
    //        }
    //    }

    private static void validateResponse(HttpResponse response, HttpPost post) throws IOException {
        StatusLine status = response.getStatusLine();
        if (status.getStatusCode() >= HttpStatus.SC_MULTIPLE_CHOICES) {
            LOG.warn(
                    "Did not receive successful HTTP response: status code = {}, status message = {}",
                    status.getStatusCode(), status.getReasonPhrase());
            post.abort();
            return;
        }
    }

}


目录
相关文章
|
存储 网络安全 对象存储
基于OSS搭建私有 Docker Registry
基于OSS搭建私有 Docker Registry Docker Registry 作为 Docker 的核心组件之一负责了镜像的存储以及分发。用户只需要使用 Docker 的客户端就可以直接和 Registry 进行交互,下载和上传镜像。
15302 43
|
开发框架 数据库 数据安全/隐私保护
FastAdmin框架实现数据表的增删改查
FastAdmin框架实现数据表的增删改查
1450 0
|
4月前
|
存储 人工智能 自然语言处理
2026 实测多AI工具协同:项目上下文统一管理落地指南
本文分享2026年实测多AI工具协同经验,提出“协同底座+专业Agent”分工模式:外部AI专注单点任务(如代码生成、日志分析),底座统一管理项目上下文、权限、版本与链路追踪。已在研报生产、代码变更、故障排障、多语言内容四大场景落地,效率提升显著,接入成本低。
|
5月前
|
数据采集 人工智能 分布式计算
多Agent集群中的"情报官"设计:为什么系统需要一个RDD
在多Agent系统中,信息采集环节的失误往往是级联错误的根源。本文从行业实践和学术研究两个维度,论证了专职情报采集Agent的必要性,并详细解析了枢衡RDD(资源探测)的五大架构设计原则,包括与CAD的对抗性协作机制等。最后提供了一套可落地的自检清单,帮助开发者判断自己的Agent集群是否需要引入专职情报官角色。
|
7月前
|
人工智能 IDE Shell
同事.Skill刷屏出圈,AI“技能蒸馏”的底层规则
本文深度解析爆火的“同事.skill”现象,揭示Skill本质是结构化提示词而非AI新能力。它通过三层渐进式披露与Harness运行时框架,解决AI工程化落地难题,而非替代人类判断力。Skill炼化的是流程,不是人。
|
8月前
|
人工智能 安全 Linux
90分钟挖出20年Linux漏洞,Claude 5.0内测曝光:软件工程正在被重写
Claude 5.0(Mythos)内测标志开发范式革命:从“人写代码”迈向“人调度AI”。它具备系统级漏洞挖掘、自主安全研究、多智能体协同等能力,推动工程师转型为任务拆解者与智能体调度者。AI不是替代,而是重构角色——人类专注定义问题与掌控系统。
|
9月前
|
数据采集 存储 人工智能
企业建设数据治理系统费用(2026年2月最新)
截至2026年,中国企业数据治理投入显著增长:2025年总支出287亿元,预计2026年超340亿元(+18.5%)。大型企业单项目平均投入860万元,中小企业多选SaaS方案(30–60万元/年)。合规驱动下,71%中企已将数据治理纳入核心IT战略。
|
人工智能 自然语言处理 物联网
从“通用AI”到“懂我AI”:企业微调专属智能助手实战指南
从“通用AI”到“懂我AI”:企业微调专属智能助手实战指南
779 9
|
11月前
|
人工智能 自然语言处理 机器人
2025年头部AI数字人公司技术解析,中国优秀数字人企业实探
2025年,数字人迈向规模化产业应用。世优科技“波塔AI数字人智能体”入选年度典型案例,凭借多模态交互、大模型+小模型架构与全栈技术,深入政务、文旅、医疗、教育等场景,实现降本增效。其技术自研、场景融合与生态共建能力,彰显数字人作为“新质生产力”的落地价值,引领人机协同新时代。
772 0
|
人工智能 自然语言处理 安全
43_PaLM与Gemma:谷歌LLM演进
在人工智能发展的浪潮中,谷歌一直扮演着关键的技术引领者角色。从最初的神经机器翻译到如今的通用人工智能,谷歌通过持续的技术创新推动着自然语言处理领域的边界不断拓展。2022年,谷歌推出了革命性的PaLM(Pathways Language Model),这一模型不仅在规模上达到了前所未有的5400亿参数,更重要的是其采用了创新的Pathways训练方法,为大型语言模型的发展开辟了新路径。随后,谷歌又推出了Gemma系列开源模型,将先进的AI技术普惠给更广泛的开发者社区。
599 2