Hadoop3.0Yarn添加网络、磁盘IO等资源资料汇总及实战配置遇到的问题和解决办法
Hadoop3.0扩展Yarn资源，官网给了一些英文资料，这里about云翻译为中文资料。这些资料，还是比较多的，但是在初次阅读的时候，会感到摸不着头脑。但是书读百遍其义自见。下面对这些资料进行汇总。
Hadoop3.0: YARN Resource自定义资源配置说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626
1.yarn默认情况下使用哪些资源？
2.Yarn如何实现扩展自定义资源？
3.自定义资源，可以在哪个配置文件中配置？
4.哪些配置可以在yarn-site.xml文件或则 resource-types.xml文件配置？
5.yarn有哪三种Containers？
6.三种Container如何配置请求的CPU，memory等值？
hadoop3.0 Yarn支持网络1：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23653
1.网络作为Yarn的资源，有什么好处？
2.Yarn是否只支持调度和强制执行“传出流量”？
3.Yarn是否支持入口流量？
hadoop3.0 Yarn支持网络资源2：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23661
1.Resource profiles解决了什么问题？
2.使用profile的好处是什么？
3.配置文件的简洁模式如何配置？
hadoop3.0扩展Yarn资源模型详解1
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23794
1.countable资源是指哪些？
2.noncountable资源，本文列举了什么资源？
3.标签是否为资源？
4.如何实现扩展YARN资源模型？
hadoop3.0扩展Yarn资源模型详解2：资源Profiles说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23813
1.Resource profiles解决了什么问题？
2.使用profile的好处是什么？
3.配置文件的简洁模式如何配置？
Hadoop3 YARN集群中的磁盘I / O调度设计详解1【IO作为资源分配】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23819
1.磁盘IO实现共享，使用的是什么原理技术？
2.如何描述磁盘I / O资源？
3.磁盘I / O资源中的调度因素如何？
4.每个NodeManager如何执行每个本地任务的磁盘I / O资源使用？
上面资料分别介绍了如何添加除了内存和vcores资源，并且对所支持的资源进行调度配置。根据以上资料，大家如果读懂，完全可以配置，如想节省时间。后面about云会出这方面的视频。
下面是在实际配置过程中遇到的问题
提交job之后，程序不运行：
hadoop jar /usr/hadoop-3.0.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar wordcount /data/wordcount /output/wordcount 2018-01-12 15:51:13,841 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.10:8032 2018-01-12 15:51:15,296 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/aboutyun/.staging/job_1515742430977_0006 2018-01-12 15:51:15,812 INFO input.FileInputFormat: Total input files to process : 1 2018-01-12 15:51:16,131 INFO mapreduce.JobSubmitter: number of splits:1 2018-01-12 15:51:16,205 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2018-01-12 15:51:16,636 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1515742430977_0006 2018-01-12 15:51:16,638 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2018-01-12 15:51:17,088 INFO conf.Configuration: found resource resource-types.xml at file:/usr/hadoop-3.0.0/etc/hadoop/resource-types.xml 2018-01-12 15:51:17,148 INFO resource.ResourceUtils: Adding resource type - name = network, units = k, type = COUNTABLE 2018-01-12 15:51:17,311 INFO impl.YarnClientImpl: Submitted application application_1515742430977_0006 2018-01-12 15:51:17,459 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1515742430977_0006/ 2018-01-12 15:51:17,460 INFO mapreduce.Job: Running job: job_1515742430977_0006
日志也没有错误，然后通过tail -f在停止集群和开启集群中，查看所有日志。原来是nodemanger的配置文件出现问题。
java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>. at [row,col {unknown-source}]: [3,68] at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3048) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689) at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2737) at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeNodeResourceInformation(ResourceUtils.java:497) at org.apache.hadoop.yarn.util.resource.ResourceUtils.getNodeResourceInformation(ResourceUtils.java:476) at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:352) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894) Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>. at [row,col {unknown-source}]: [3,68] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475) at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365) at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2883) ... 13 more 2018-01-12 15:57:07,441 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10 ************************************************************/ ^C [aboutyun@master logs]$ tail -f hadoop-aboutyun-nodemanager-master.log at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365) at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2883) ... 13 more 2018-01-12 15:57:07,441 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
去掉里面的错误标记问题解决。
yarn-site.xml文件
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value> </property> <property> <name>yarn.nodemanager.vmem-check-enable</name> <value>false</value> </property> <property> <name>yarn.scheduler.enablenetworkscheduling</name> <value>enable</value> </property> <property> <name>yarn.scheduler.minimumallocationoutboundnetworkbandwidthmbit</name> <value>1mbit/sec</value> </property> <property> <name>yarn.scheduler.maximumallocationoutboundnetworkbandwidthmbit</name> <value>1000mbit/sec</value> </property> <property> <name>yarn.nodemanager.resource.network.interface</name> <value>eno16777736</value> </property> <property> <name>yarn.nodemanager.resource.network.outboundbandwidthmbit</name> <value>1000mbit/sec</value> </property> <property> <name>yarn.nodemanager.resource.network.outboundbandwidthyarnmbit</name> <value>1000mbit/sec</value> </property> <property> <name>yarn.nodemanager.strictresourceusag</name> <value>elastic</value> </property> </configuration>
下面说是network是未知的资源network：
2018-01-19 17:16:36,843 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.ResourceNotFoundException: Unknown resource 'network'. Known resources are [name: memory-mb, units: Mi, type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807, name: vcores, units: , type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807] at org.apache.hadoop.yarn.api.records.Resource.getResourceInformation(Resource.java:251) at org.apache.hadoop.yarn.api.records.Resource.setResourceInformation(Resource.java:311) at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:355) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894) 2018-01-19 17:16:36,934 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10
上面是因为没有弄懂如何添加网络资源，需要区分开与memory和vcore的区别。需仔细阅读
Hadoop3.0: YARN Resource自定义资源配置说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626并且区分
hadoop3.0 Yarn支持网络1：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23653
这个文档的关系。
2018-01-20 10:04:54,240 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'. at org.apache.hadoop.yarn.util.resource.ResourceUtils.checkMandatoryResources(ResourceUtils.java:94) at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourcesMap(ResourceUtils.java:276) at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:382) at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:366) at org.apache.hadoop.yarn.util.resource.ResourceUtils.getNumberOfKnownResourceTypes(ResourceUtils.java:354) at org.apache.hadoop.yarn.api.records.impl.LightWeightResource.<init>(LightWeightResource.java:70) at org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:79) at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:350) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)
出现上面问题，同样也是因为没有弄懂如何添加资源及调度的关系。
