开发者社区> 问答> 正文

E-MapReduce中遇到ERROR cluster.YarnScheduler: Lost executor Container killed by YARN for exceeding memory limits

已解决

作业运行失败。提示错误信息包含
ERROR cluster.YarnScheduler: Lost executor Container killed by YARN for exceeding memory limits

展开
收起
计算爱好者 2016-08-16 20:44:06 12788 0
1 条回答
写回答
取消 提交回答
  • 采纳回答

    该错误会跟随这样的提示信息 Consider boosting spark.yarn.executor.memoryOverhead.
    按着原来就可以了。调大这个参数的值。默认情况 spark.yarn.executor.memoryOverhead executorMemory * 0.10, with minimum of 384
    spark 参数加上 --conf spark.yarn.executor.memoryOverhead=512 或者--conf spark.yarn.executor.memoryOverhead=768 或者 --conf spark.yarn.executor.memoryOverhead=1024 ,调大这个参数去解决你的问题。

    2019-07-17 20:03:59
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
How to overcome mysterious problems caused by large and multi-tenant hadoop cluster at Rakuten 立即下载
Sparklint a Tool for Identifying and Tuning Inefficient Spark Jobs Across Your Cluster 立即下载
ALLUXIO (FORMERLY TACHYON): UNIFY DATA AT MEMORY SPEED - EFFECTIVE USING SPARK WITH ALLUXIO 立即下载