Hadoop3.0Yarn添加网络、磁盘IO等资源资料汇总及实战配置遇到的问题和解决办法

Hadoop3.0扩展Yarn资源，官网给了一些英文资料，这里about云翻译为中文资料。这些资料，还是比较多的，但是在初次阅读的时候，会感到摸不着头脑。但是书读百遍其义自见。下面对这些资料进行汇总。

Hadoop3.0: YARN Resource自定义资源配置说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626
1.yarn默认情况下使用哪些资源？
2.Yarn如何实现扩展自定义资源？
3.自定义资源，可以在哪个配置文件中配置？
4.哪些配置可以在yarn-site.xml文件或则 resource-types.xml文件配置？
5.yarn有哪三种Containers？
6.三种Container如何配置请求的CPU，memory等值？



hadoop3.0 Yarn支持网络1：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23653
1.网络作为Yarn的资源，有什么好处？
2.Yarn是否只支持调度和强制执行“传出流量”？
3.Yarn是否支持入口流量？


hadoop3.0 Yarn支持网络资源2：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23661
1.Resource profiles解决了什么问题？
2.使用profile的好处是什么？
3.配置文件的简洁模式如何配置？




hadoop3.0扩展Yarn资源模型详解1
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23794
1.countable资源是指哪些？
2.noncountable资源，本文列举了什么资源？
3.标签是否为资源？
4.如何实现扩展YARN资源模型？



hadoop3.0扩展Yarn资源模型详解2：资源Profiles说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23813
1.Resource profiles解决了什么问题？
2.使用profile的好处是什么？
3.配置文件的简洁模式如何配置？



Hadoop3 YARN集群中的磁盘I / O调度设计详解1【IO作为资源分配】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23819
1.磁盘IO实现共享，使用的是什么原理技术？
2.如何描述磁盘I / O资源？
3.磁盘I / O资源中的调度因素如何？
4.每个NodeManager如何执行每个本地任务的磁盘I / O资源使用？


上面资料分别介绍了如何添加除了内存和vcores资源，并且对所支持的资源进行调度配置。根据以上资料，大家如果读懂，完全可以配置，如想节省时间。后面about云会出这方面的视频。
下面是在实际配置过程中遇到的问题：
提交job之后，程序不运行：
[mw_shl_code=bash,true]hadoop jar /usr/hadoop-3.0.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar wordcount /data/wordcount /output/wordcount
2018-01-12 15:51:13,841 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.10:8032
2018-01-12 15:51:15,296 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/aboutyun/.staging/job_1515742430977_0006
2018-01-12 15:51:15,812 INFO input.FileInputFormat: Total input files to process : 1
2018-01-12 15:51:16,131 INFO mapreduce.JobSubmitter: number of splits:1
2018-01-12 15:51:16,205 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-01-12 15:51:16,636 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1515742430977_0006
2018-01-12 15:51:16,638 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-01-12 15:51:17,088 INFO conf.Configuration: found resource resource-types.xml at file:/usr/hadoop-3.0.0/etc/hadoop/resource-types.xml
2018-01-12 15:51:17,148 INFO resource.ResourceUtils: Adding resource type - name = network, units = k, type = COUNTABLE
2018-01-12 15:51:17,311 INFO impl.YarnClientImpl: Submitted application application_1515742430977_0006
2018-01-12 15:51:17,459 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1515742430977_0006/
2018-01-12 15:51:17,460 INFO mapreduce.Job: Running job: job_1515742430977_0006[/mw_shl_code]

日志也没有错误，然后通过tail -f在停止集群和开启集群中，查看所有日志。原来是nodemanger的配置文件出现问题。
[mw_shl_code=bash,true]java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>.
at [row,col {unknown-source}]: [3,68]
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3048)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
        at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2737)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeNodeResourceInformation(ResourceUtils.java:497)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.getNodeResourceInformation(ResourceUtils.java:476)
        at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:352)
        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>.
at [row,col {unknown-source}]: [3,68]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
        at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
        at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2883)
        ... 13 more
2018-01-12 15:57:07,441 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10
************************************************************/
^C
[aboutyun@master logs]$ tail -f hadoop-aboutyun-nodemanager-master.log
        at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
        at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2883)
        ... 13 more
2018-01-12 15:57:07,441 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:[/mw_shl_code]

去掉里面的错误标记问题解决。



yarn-site.xml文件

[mw_shl_code=bash,true]<configuration>

<!-- Site specific YARN configuration properties -->
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
   <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enable</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.scheduler.enablenetworkscheduling</name>
        <value>enable</value>
    </property>
<property>
        <name>yarn.scheduler.minimumallocationoutboundnetworkbandwidthmbit</name>
        <value>1mbit/sec</value>
    </property>
<property>
        <name>yarn.scheduler.maximumallocationoutboundnetworkbandwidthmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.interface</name>
        <value>eno16777736</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.outboundbandwidthmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.outboundbandwidthyarnmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.strictresourceusag</name>
        <value>elastic</value>
    </property>


</configuration>[/mw_shl_code]


下面说是network是未知的资源network：
[mw_shl_code=bash,true]2018-01-19 17:16:36,843 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.ResourceNotFoundException: Unknown resource 'network'. Known resources are [name: memory-mb, units: Mi, type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807, name: vcores, units: , type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807]
        at org.apache.hadoop.yarn.api.records.Resource.getResourceInformation(Resource.java:251)
        at org.apache.hadoop.yarn.api.records.Resource.setResourceInformation(Resource.java:311)
        at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:355)
        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)
2018-01-19 17:16:36,934 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10[/mw_shl_code]

上面是因为没有弄懂如何添加网络资源，需要区分开与memory和vcore的区别。需仔细阅读
Hadoop3.0: YARN Resource自定义资源配置说明
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626并且区分
hadoop3.0 Yarn支持网络1：network设计文档说明【中文】
http://www.aboutyun.com/forum.php?mod=viewthread&tid=23653
这个文档的关系。

[mw_shl_code=bash,true]
2018-01-20 10:04:54,240 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'.
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.checkMandatoryResources(ResourceUtils.java:94)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourcesMap(ResourceUtils.java:276)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:382)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:366)
        at org.apache.hadoop.yarn.util.resource.ResourceUtils.getNumberOfKnownResourceTypes(ResourceUtils.java:354)
        at org.apache.hadoop.yarn.api.records.impl.LightWeightResource.<init>(LightWeightResource.java:70)
        at org.apache.hadoop.yarn.api.records.Resource.newInstance(Resource.java:79)
        at org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:350)
        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)[/mw_shl_code]

出现上面问题，同样也是因为没有弄懂如何添加资源及调度的关系。