Hadoop必看1-2：CentOS Hadoop-2.2.0集群安装配置实践

继续接着上一篇内容
Hadoop必看1-1：Hadoop-2.2.0集群安装配置实践

1.如何启动集群2.如何通过web查看集群
3.查看查看slave资源状态？
4.JobHistory Server的作用是什么？
5.如何验证wordcount?

启动YARN集群
在主节点m1上，执行如下命令：

start-yarn.sh

复制代码

可以查看启动日志，确认YARN集群启动是否成功：

tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-resourcemanager-m1.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s1.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s2.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s3.log

复制代码

或者，查看对应的进程情况：

  jps

复制代码

另外，ResourceManager运行在主节点m1上，可以Web控制台查看状态：

 http://m1:8088/

复制代码

NodeManager运行在从节点上，可以通过Web控制台查看对应节点的资源状态，例如节点s1：

http://s1:8042/

复制代码

管理JobHistory Server
启动可以JobHistory Server，能够通过Web控制台查看集群计算的任务的信息，执行如下命令：

mr-jobhistory-daemon.sh start historyserver

复制代码

默认使用19888端口。
通过访问http://m1:19888/查看任务执行历史信息。
终止JobHistory Server，执行如下命令：

mr-jobhistory-daemon.sh stop historyserver

复制代码

集群验证
我们使用Hadoop自带的WordCount例子进行验证。
先在HDFS创建几个数据目录：

hadoop fs -mkdir -p /data/wordcount
hadoop fs -mkdir -p /output/

复制代码

目录/data/wordcount用来存放Hadoop自带的WordCount例子的数据文件，运行这个MapReduce任务的结果输出到/output/wordcount目录中。
将本地文件上传到HDFS中：

hadoop fs -put /home/shirdrn/cloud/programs/hadoop-2.2.0/etc/hadoop/*.xml /data/wordcount/
复制代码

可以查看上传后的文件情况，执行如下命令：

hadoop fs -ls /data/wordcount

复制代码

可以看到上传到HDFS中的文件。
下面，运行WordCount例子，执行如下命令：

hadoop jar /home/shirdrn/cloud/programs/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
复制代码

可以看到控制台输出程序运行的信息：

[shirdrn@m1 hadoop-2.2.0]$ hadoop jar /home/shirdrn/cloud/programs/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
13/12/25 22:38:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/25 22:38:03 INFO client.RMProxy: Connecting to ResourceManager at m1/10.95.3.48:8032
13/12/25 22:38:04 INFO input.FileInputFormat: Total input paths to process : 7
13/12/25 22:38:04 INFO mapreduce.JobSubmitter: number of splits:7
13/12/25 22:38:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/12/25 22:38:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1388039619930_0002
13/12/25 22:38:05 INFO impl.YarnClientImpl: Submitted application application_1388039619930_0002 to ResourceManager at m1/10.95.3.48:8032
13/12/25 22:38:05 INFO mapreduce.Job: The url to track the job: http://m1:8088/proxy/application_1388039619930_0002/
13/12/25 22:38:05 INFO mapreduce.Job: Running job: job_1388039619930_0002
13/12/25 22:38:14 INFO mapreduce.Job: Job job_1388039619930_0002 running in uber mode : false
13/12/25 22:38:14 INFO mapreduce.Job:  map 0% reduce 0%
13/12/25 22:38:22 INFO mapreduce.Job:  map 14% reduce 0%
13/12/25 22:38:42 INFO mapreduce.Job:  map 29% reduce 5%
13/12/25 22:38:43 INFO mapreduce.Job:  map 43% reduce 5%
13/12/25 22:38:45 INFO mapreduce.Job:  map 43% reduce 14%
13/12/25 22:38:54 INFO mapreduce.Job:  map 57% reduce 14%
13/12/25 22:38:55 INFO mapreduce.Job:  map 71% reduce 19%
13/12/25 22:38:56 INFO mapreduce.Job:  map 100% reduce 19%
13/12/25 22:38:57 INFO mapreduce.Job:  map 100% reduce 100%
13/12/25 22:38:58 INFO mapreduce.Job: Job job_1388039619930_0002 completed successfully
13/12/25 22:38:58 INFO mapreduce.Job: Counters: 44
     File System Counters
          FILE: Number of bytes read=15339
          FILE: Number of bytes written=667303
          FILE: Number of read operations=0
          FILE: Number of large read operations=0
          FILE: Number of write operations=0
          HDFS: Number of bytes read=21904
          HDFS: Number of bytes written=9717
          HDFS: Number of read operations=24
          HDFS: Number of large read operations=0
          HDFS: Number of write operations=2
     Job Counters
          Killed map tasks=2
          Launched map tasks=9
          Launched reduce tasks=1
          Data-local map tasks=9
          Total time spent by all maps in occupied slots (ms)=457338
          Total time spent by all reduces in occupied slots (ms)=65832
     Map-Reduce Framework
          Map input records=532
          Map output records=1923
          Map output bytes=26222
          Map output materialized bytes=15375
  Input split bytes=773
          Combine input records=1923
          Combine output records=770
          Reduce input groups=511
          Reduce shuffle bytes=15375
          Reduce input records=770
          Reduce output records=511
          Spilled Records=1540
          Shuffled Maps =7
          Failed Shuffles=0
          Merged Map outputs=7
          GC time elapsed (ms)=3951
          CPU time spent (ms)=22610
          Physical memory (bytes) snapshot=1598832640
          Virtual memory (bytes) snapshot=6564274176
          Total committed heap usage (bytes)=971993088
     Shuffle Errors
          BAD_ID=0
          CONNECTION=0
          IO_ERROR=0
          WRONG_LENGTH=0
          WRONG_MAP=0
          WRONG_REDUCE=0
     File Input Format Counters
          Bytes Read=21131
     File Output Format Counters
          Bytes Written=9717

复制代码

查看结果，执行如下命令：

hadoop fs -cat /output/wordcount/part-r-00000 | head
复制代码

结果数据示例如下：

[shirdrn@m1 hadoop-2.2.0]$ hadoop fs -cat /output/wordcount/part-r-00000 | head
13/12/25 22:58:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
"*"     17
"AS     3
"License");     3
"alice,bob     17
$HADOOP_HOME/share/hadoop/common/lib/*,     1
$HADOOP_HOME/share/hadoop/hdfs/*,$HADOOP_HOME/share/hadoop/hdfs/lib/*,     1
$HADOOP_HOME/share/hadoop/mapreduce/*,$HADOOP_HOME/share/hadoop/mapreduce/lib/*</value>     1
$HADOOP_HOME/share/hadoop/yarn/*,$HADOOP_HOME/share/hadoop/yarn/lib/*,     1
(ASF)     1
(YARN-1229)-->     1
cat: Unable to write to output stream.

复制代码

登录到Web控制台，访问链接http://m1:8088/可以看到任务记录情况。
可见，我们的HDFS能够存储数据，而YARN集群也能够运行MapReduce任务。

问题及总结

需要知道的默认配置

在Hadoop 2.2.0中，YARN框架有很多默认的参数值，如果你是在机器资源比较不足的情况下，需要修改这些默认值，来满足一些任务需要。

NodeManager和ResourceManager都是在yarn-site.xml文件中配置的，而运行MapReduce任务时，是在mapred-site.xml中进行配置的。

下面看一下相关的参数及其默认值情况：

参数名称	默认值	进程名称	配置文件	含义说明
yarn.nodemanager.resource.memory-mb	8192	NodeManager	yarn-site.xml	从节点所在物理主机的可用物理内存总量
yarn.nodemanager.resource.cpu-vcores	8	NodeManager	yarn-site.xml	节点所在物理主机的可用虚拟CPU资源总数（core）
yarn.nodemanager.vmem-pmem-ratio	2.1	NodeManager	yarn-site.xml	使用1M物理内存，最多可以使用的虚拟内存数量
yarn.scheduler.minimum-allocation-mb	1024	ResourceManager	yarn-site.xml	一次申请分配内存资源的最小数量
yarn.scheduler.maximum-allocation-mb	8192	ResourceManager	yarn-site.xml	一次申请分配内存资源的最大数量
yarn.scheduler.minimum-allocation-vcores	1	ResourceManager	yarn-site.xml	一次申请分配虚拟CPU资源最小数量
yarn.scheduler.maximum-allocation-vcores	8	ResourceManager	yarn-site.xml	一次申请分配虚拟CPU资源最大数量
mapreduce.framework.name	local	MapReduce	mapred-site.xml	取值local、classic或yarn其中之一，如果不是yarn，则不会使用YARN集群来实现资源的分配
mapreduce.map.memory.mb	1024	MapReduce	mapred-site.xml	每个MapReduce作业的map任务可以申请的内存资源数量
mapreduce.map.cpu.vcores	1	MapReduce	mapred-site.xml	每个MapReduce作业的map任务可以申请的虚拟CPU资源的数量
mapreduce.reduce.memory.mb	1024	MapReduce	mapred-site.xml	每个MapReduce作业的reduce任务可以申请的内存资源数量
yarn.nodemanager.resource.cpu-vcores	8	MapReduce	mapred-site.xml	每个MapReduce作业的reduce任务可以申请的虚拟CPU资源的数量

异常java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.66:50010

详细异常信息，如下所示：

[shirdrn@m1 hadoop-2.2.0]$ hadoop fs -put /home/shirdrn/cloud/programs/hadoop-2.2.0/etc/hadoop/*.xml /data/wordcount/
13/12/25 21:29:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.66:50010
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1166)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-1388035628061:blk_1073741825_1001
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.66:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.59:50010
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1166)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-1388035628061:blk_1073741826_1002
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.59:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
     at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1128)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-
1388035628061:blk_1073741828_1004
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.59:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
复制代码