连接Zookeeper , ConnLoss 导致OOM

依赖版本

zookeeper: 3.4.9

    <dependency>
        <groupId>org.apache.dubbo</groupId>
        <artifactId>dubbo-spring-boot-starter</artifactId>
        <version>2.7.8</version>
    </dependency>

    <dependency>
        <groupId>org.apache.curator</groupId>
        <artifactId>curator-recipes</artifactId>
        <version>2.10.0</version>
    </dependency>

dubbo配置

dubbo.protocol.lazy=true dubbo.protocol.heartbeat=3000 dubbo.registry.address= dubbo.registry.register=false dubbo.consumer.check=false dubbo.application.qos.enable=false dubbo.registry.file=./dubboregistry/dubbo-registry.properties dubbo.provider.timeout=10000 dubbo.provider.threadpool=fixed dubbo.provider.threads=100 dubbo.provider.protocol=dubbo dubbo.provider.cluster=failfast dubbo.provider.loadbalance=roundrobin dubbo.provider.server=netty dubbo.consumer.lazy=true dubbo.consumer.timeout=10000 dubbo.application.logger=slf4j

遇到的错误日志如下：

2021-11-11 09:49:40 | siac | * | Curator-Framework-0 | INFO | o.a.curator.framework.state.ConnectionStateManager | | ConnectionStateManager.java:228 | State change: LOST 2021-11-11 09:49:41 | siac | * | Curator-Framework-0 | INFO | o.apache.curator.framework.recipes.cache.TreeCache | | TreeCache.java:469 | Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', name='null', children=null, context=null, stat=null, data=null, watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null} 2021-11-11 09:49:41 | siac | * | Curator-Framework-0 | INFO | o.apache.curator.framework.recipes.cache.TreeCache | | TreeCache.java:469 | Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', name='null', children=null, context=null, stat=null, data=null, watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null}

2021-11-11 09:49:48 | siac | * | DubboRegistryRetryTimer-thread-1 | INFO | o.apache.dubbo.registry.retry.FailedSubscribedTask | | AbstractRetryTask.java:121 | [DUBBO] retry subscribe : consumer://10.135.101.104/com.authority.service.IAuthorityService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.authority.service.IAuthorityService&lazy=true&logger=slf4j&metadata-type=remote&methods=getRoleInfoByRoleEnName,updateRole,addResources,getAuthorithByUsername,deleteResource,getOperateLogList,addLog,disableRole,getUsefulRoles,getResourcesDetail,batchInsertAuthUserRole,getRoles,powerList,getResourcesTree,getUserByUsername,getResourcesList,getRole,getAuthTreeByRoleId,selectBdList,getUserInfoByRoleId,getRoleByUsername,modifyResource,getUserInfoByRoleEnName,powerRedis,userRole,addRole,getRolesByResKey&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446226827, dubbo version: 2.7.8, current host: 10.135.101.104 2021-11-11 09:49:48 | siac | * | Curator-Framework-0 | INFO | o.apache.curator.framework.recipes.cache.TreeCache | | TreeCache.java:469 | Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', name='null', children=null, context=null, stat=null, data=null, watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null}

2021-11-11 09:49:47 | siac | * | main-SendThread(10.135.101.101:2181) | WARN | org.apache.zookeeper.ClientCnxn | | ClientCnxn.java:1102 | Session 0x17a78d8154236ec for server 10.135.101.101/10.135.101.101:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2021-11-11 09:49:47 | siac | * | Curator-ConnectionStateManager-0 | WARN | o.a.d.r.zookeeper.curator.CuratorZookeeperClient | | CuratorZookeeperClient.java:376 | [DUBBO] Curator zookeeper connection recovered from connection lose, reuse the old session 17a78d8154236ec, dubbo version: 2.7.8, current host: 10.135.101.104 2021-11-11 09:49:47 | siac | * | Curator-ConnectionStateManager-0 | WARN | o.a.dubbo.registry.zookeeper.ZookeeperRegistry | | ZookeeperRegistry.java:89 | [DUBBO] Trying to fetch the latest urls, in case there're provider changes during connection loss. Since ephemeral ZNode will not get deleted for a connection lose, there's no need to re-register url of this instance., dubbo version: 2.7.8, current host: 10.135.101.104

2021-11-11 10:23:46 | siac | * | main-EventThread | ERROR | o.a.curator.framework.imps.CuratorFrameworkImpl | | CuratorFrameworkImpl.java:557 | Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516) at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:593) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

内存溢出的直接原因在：

绝大多数的 ClientCnxn$Packet 的 AsyncCallback cb 为 GetChildrenBuilderImpl。

存活对象：

遇到过两次这种问题。

原提问者GitHub用户shenhuaxin