由于写入ZK节点数据过大限制导致RM切换失败的问题该怎么复现？

这个问题很难复现，前两次一直没找到产生该问题的原因，打了patch之后，我们在日志中发现，产生该问题主要是由于部分异常任务导致的，日志如下： 2020-04-28 10:05:54 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:768 - Application update attemptState data size for /rmstore/ZKRMStateRoot/RMAppRoot/application_1587969707206_16259/appattempt_1587969707206_16259_000001 is 20266528. Exceed the maximum allowed 3145728 size. ApplicationAttemptState info: ApplicationAttemptState{attemptId=appattempt_1587969707206_16259_000001, diagnostics='User class threw exception: java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 15 in stage 2.0 failed 4 times, most recent failure: Lost task 15.3 in stage 2.0 (TID 4224, datanode44.bi): java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB /* 001 / public java.lang.Object generate(Object[] references) { / 002 / return new SpecificUnsafeProjection(references); / 003 / } / 004 / / 005 / class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { / 006 / / 007 / private Object[] references; / 008 / private scala.collection.immutable.Set hset; / 009 / private boolean hasNull; / 010 / private UnsafeRow result; / 011 / private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder holder; / 012 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter; 当任务出现异常时，YARN会保存任务的异常信息，当异常信息很多时，YARN往ZK保存任务状态的数据量就会超过ZK的限制。从日志中可以看出，出现异常的Spark任务状态数据是20266528字节，也就是19MB，远远超过了我们所设置的3MB。 由于有了前两次发现和解决问题以及源码理解的经验，所以这次解决问题就顺手的多，去年八月份解决该问题的最终方案是调整ZK服务端和YARN客户端的jute.maxbuffer参数值为3MB，也就是调整ZK中每个ZNode能保存的最大数据量为3MB。 原文链接：https://blog.csdn.net/github_32521685/article/details/106052793