背景介绍
NSF(Netease Service Framework)是网易数帆下的一款微服务框架，项目在压测过程中，发现NSF FeignClient有性能瓶颈，下面对遇到的问题及优化方案进行分析，以备忘。
问题分析
在压测过程中发现系统耗时高，通过arthas thread -b发现大量线程处于阻塞状态，进一步分析发现NSF FeignClient使用的java.net.HttpURLConnection，而NSF FeignClient当前版本没有方便的方式（比如改个配置）来支持其他HttpClient，以下通过对NSF FeignClient实现分析来尝试解决这个问题。
NSF FeignClient实现方式
NSF是通过nsf-agent将自定义的FeignClient注入到Spring容器中的，下面是NSF FeignClient的一些主要实现逻辑。
SpringConfigurationTransformer
该类向Spring注入了一些Configuration组件。
public class SpringConfigurationTransformer implements NoneNamedTransformer { private static Logger logger = AgentLogFactory.getLogger(SpringConfigurationTransformer.class); public TransformerResult transform(ClassLoader paramClassLoader, String paramString, Class<?> paramClass, ProtectionDomain paramProtectionDomain, byte[] paramArrayOfByte, Object paramObject) throws Exception { ClassPool classPool = ClassPool.getDefault(); InputStream ins = new ByteArrayInputStream(paramArrayOfByte); CtClass ctclass = classPool.makeClass(ins); TransformerResult result = new TransformerResult(ctclass); result.setClassModified(Boolean.valueOf(true)); logger.debug("nsf-agent: SpringConfigurationTransformer process transform ..."); List<String> importConfigurationList = new ArrayList<>(); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringContextConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NsfRegistryConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringFeignConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringRibbonConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.ServletFilterConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.ToolsConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NacosToolsConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.core.autoconfig.ConfigDebugConfiguration"); importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NsfSpringDiscoveryClientConfiguration"); JavassistUtil.addImportAnnotations(ctclass, importConfigurationList); result.setClassCodeBytes(ctclass.toBytecode()); ctclass.defrost(); return result; } public Object classNameMatch(String paramString) { if (paramString.equalsIgnoreCase(GlobalConfigManager.getSpringBootClassName())) return Boolean.valueOf(true); return null; } }
与该问题直接相关的是SpringFeignConfiguration。
SpringFeignConfiguration
将自定义的feignClient相关配置注入Spring。
@Configuration @ConditionalOnClass(name = {"feign.Client", "org.springframework.cloud.openfeign.FeignClient"}) public class SpringFeignConfiguration { private Logger logger = AgentLogFactory.getLogger(SpringFeignConfiguration.class); @PostConstruct public void initFeign() { this.logger.info("find feign, init NsfFeignClient ..."); } @Bean(name = {"feignClient"}) @ConditionalOnClass(name = {"org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient"}) public Client loadBalancerFeignClient(RestTemplate nsfMethodRestTemplate) { return (Client)new NsfLoadBalancerFeignClient(nsfMethodRestTemplate); } @Bean(name = {"feignClient"}) @ConditionalOnMissingClass({"org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient"}) public NsfFeignClient feignClient(RestTemplate nsfMethodRestTemplate) { return new NsfFeignClient(nsfMethodRestTemplate); } }
在运行环境中，加载的是NsfLoadBalancerFeignClient。
NsfLoadBalancerFeignClient
public class NsfLoadBalancerFeignClient extends LoadBalancerFeignClient implements Client { private NsfFeignClient feignClient; public NsfLoadBalancerFeignClient(RestTemplate restTemplate) { super((Client)new Client.Default(null, null), null, null); this.feignClient = new NsfFeignClient(restTemplate); } public Response execute(Request request, Request.Options options) throws IOException { return this.feignClient.execute(request, options); } }
我们接着查看NsfFeignClient的实现。
NsfFeignClient
public class NsfFeignClient implements Client { private static final String ACCEPT_HEADER_NAME = "Accept"; private ClientHttpRequestFactory requestFactory; public NsfFeignClient(RestTemplate restTemplate) { this.requestFactory = (ClientHttpRequestFactory)new InterceptingClientHttpRequestFactory((ClientHttpRequestFactory)new NsfClientHttpRequestFactory(), restTemplate.getInterceptors()); } ... ... }
其中requestFactory是通过NsfClientHttpRequestFactory进行初始化的，也是问题的关键所在，下面看下NsfClientHttpRequestFactory的实现。
NsfClientHttpRequestFactory
public class NsfClientHttpRequestFactory extends SimpleClientHttpRequestFactory { public static final ThreadLocal<TimeOut> timeoutCache = new ThreadLocal<TimeOut>() { protected NsfClientHttpRequestFactory.TimeOut initialValue() { return new NsfClientHttpRequestFactory.TimeOut(); } }; ... ... }
NsfClientHttpRequestFactory继承了Spring中的SimpleClientHttpRequestFactory，下面看下SimpleClientHttpRequestFactory中创建连接的逻辑。
SimpleClientHttpRequestFactory
@Override public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException { HttpURLConnection connection = openConnection(uri.toURL(), this.proxy); prepareConnection(connection, httpMethod.name()); if (this.bufferRequestBody) { return new SimpleBufferingClientHttpRequest(connection, this.outputStreaming); } else { return new SimpleStreamingClientHttpRequest(connection, this.chunkSize, this.outputStreaming); } } protected HttpURLConnection openConnection(URL url, @Nullable Proxy proxy) throws IOException { URLConnection urlConnection = (proxy != null ? url.openConnection(proxy) : url.openConnection()); if (!(urlConnection instanceof HttpURLConnection)) { throw new IllegalStateException( "HttpURLConnection required for [" + url + "] but got: " + urlConnection); } return (HttpURLConnection) urlConnection; }
其中创建连接的逻辑在openConnection方法中，使用HttpURLConnection创建的连接。
优化方案
通过问题分析知道了问题所在，优化方案就比较简单了，只要想办法替换掉java.net.HttpURLConnection就可以了，为了尽可能小的影响NSF FeignClient已有逻辑，我们仿照NSF定义一套相应的类，只需要替换掉java.net.HttpURLConnection的逻辑就可以了。
自定义FeignClient
仿照现有NSF FeignClient单独写一套即可，关键是替换掉java.net.HttpURLConnection。
替换NSF feignClient
通过BeanDefinitionRegistry使用我们自定义的feignClient替换掉NSF feignClient。
public class DefaultBeanDefinitionRegistryPostProcessor implements BeanDefinitionRegistryPostProcessor { @Override public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry beanDefinitionRegistry) throws BeansException { String[] beanDefinitionNames = beanDefinitionRegistry.getBeanDefinitionNames(); for(String beanName : beanDefinitionNames){ if(beanName.equals("feignClient")){ BeanDefinition wlFeignClient = beanDefinitionRegistry.getBeanDefinition("wlFeignClient"); if(wlFeignClient != null) { beanDefinitionRegistry.removeBeanDefinition("feignClient"); beanDefinitionRegistry.removeBeanDefinition("wlFeignClient"); beanDefinitionRegistry.registerBeanDefinition("feignClient", wlFeignClient); } break; } } } @Override public void postProcessBeanFactory(ConfigurableListableBeanFactory configurableListableBeanFactory) throws BeansException { } }
总结
通过arthas定位出问题原因，通过分析搞清楚NSF FeignClient的实现，然后再实现一套支持Apache HttpClient的方案来替换原来的HttpURLConnection，经过测试NSF FeignClient已经不再成为阻塞点了。
