Hadoop必看1-2：CentOS Hadoop-2.2.0集群安装配置实践
继续接着上一篇内容Hadoop必看1-1：Hadoop-2.2.0集群安装配置实践
1.如何启动集群2.如何通过web查看集群
3.查看查看slave资源状态？
4.JobHistory Server的作用是什么？
5.如何验证wordcount?
static/image/hrline/4.gif
启动YARN集群
在主节点m1上，执行如下命令：
start-yarn.sh
可以查看启动日志，确认YARN集群启动是否成功：
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-resourcemanager-m1.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s1.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s2.log
tail -100f /home/shirdrn/cloud/storage/hadoop-2.2.0/logs/yarn-shirdrn-nodemanager-s3.log
或者，查看对应的进程情况：
jps
另外，ResourceManager运行在主节点m1上，可以Web控制台查看状态：
http://m1:8088/
NodeManager运行在从节点上，可以通过Web控制台查看对应节点的资源状态，例如节点s1：
http://s1:8042/
管理JobHistory Server
启动可以JobHistory Server，能够通过Web控制台查看集群计算的任务的信息，执行如下命令：
mr-jobhistory-daemon.sh start historyserver
默认使用19888端口。
通过访问http://m1:19888/查看任务执行历史信息。
终止JobHistory Server，执行如下命令：
mr-jobhistory-daemon.sh stop historyserver
集群验证
我们使用Hadoop自带的WordCount例子进行验证。
先在HDFS创建几个数据目录：
hadoop fs -mkdir -p /data/wordcount
hadoop fs -mkdir -p /output/
目录/data/wordcount用来存放Hadoop自带的WordCount例子的数据文件，运行这个MapReduce任务的结果输出到/output/wordcount目录中。
将本地文件上传到HDFS中：
hadoop fs -put /home/shirdrn/cloud/programs/hadoop-2.2.0/etc/hadoop/*.xml /data/wordcount/
可以查看上传后的文件情况，执行如下命令：
hadoop fs -ls /data/wordcount
可以看到上传到HDFS中的文件。
下面，运行WordCount例子，执行如下命令：
hadoop jar /home/shirdrn/cloud/programs/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
可以看到控制台输出程序运行的信息：
$ hadoop jar /home/shirdrn/cloud/programs/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
13/12/25 22:38:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/25 22:38:03 INFO client.RMProxy: Connecting to ResourceManager at m1/10.95.3.48:8032
13/12/25 22:38:04 INFO input.FileInputFormat: Total input paths to process : 7
13/12/25 22:38:04 INFO mapreduce.JobSubmitter: number of splits:7
13/12/25 22:38:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/12/25 22:38:04 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/12/25 22:38:04 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/12/25 22:38:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1388039619930_0002
13/12/25 22:38:05 INFO impl.YarnClientImpl: Submitted application application_1388039619930_0002 to ResourceManager at m1/10.95.3.48:8032
13/12/25 22:38:05 INFO mapreduce.Job: The url to track the job: http://m1:8088/proxy/application_1388039619930_0002/
13/12/25 22:38:05 INFO mapreduce.Job: Running job: job_1388039619930_0002
13/12/25 22:38:14 INFO mapreduce.Job: Job job_1388039619930_0002 running in uber mode : false
13/12/25 22:38:14 INFO mapreduce.Job:map 0% reduce 0%
13/12/25 22:38:22 INFO mapreduce.Job:map 14% reduce 0%
13/12/25 22:38:42 INFO mapreduce.Job:map 29% reduce 5%
13/12/25 22:38:43 INFO mapreduce.Job:map 43% reduce 5%
13/12/25 22:38:45 INFO mapreduce.Job:map 43% reduce 14%
13/12/25 22:38:54 INFO mapreduce.Job:map 57% reduce 14%
13/12/25 22:38:55 INFO mapreduce.Job:map 71% reduce 19%
13/12/25 22:38:56 INFO mapreduce.Job:map 100% reduce 19%
13/12/25 22:38:57 INFO mapreduce.Job:map 100% reduce 100%
13/12/25 22:38:58 INFO mapreduce.Job: Job job_1388039619930_0002 completed successfully
13/12/25 22:38:58 INFO mapreduce.Job: Counters: 44
File System Counters
FILE: Number of bytes read=15339
FILE: Number of bytes written=667303
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=21904
HDFS: Number of bytes written=9717
HDFS: Number of read operations=24
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Killed map tasks=2
Launched map tasks=9
Launched reduce tasks=1
Data-local map tasks=9
Total time spent by all maps in occupied slots (ms)=457338
Total time spent by all reduces in occupied slots (ms)=65832
Map-Reduce Framework
Map input records=532
Map output records=1923
Map output bytes=26222
Map output materialized bytes=15375
Input split bytes=773
Combine input records=1923
Combine output records=770
Reduce input groups=511
Reduce shuffle bytes=15375
Reduce input records=770
Reduce output records=511
Spilled Records=1540
Shuffled Maps =7
Failed Shuffles=0
Merged Map outputs=7
GC time elapsed (ms)=3951
CPU time spent (ms)=22610
Physical memory (bytes) snapshot=1598832640
Virtual memory (bytes) snapshot=6564274176
Total committed heap usage (bytes)=971993088
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=21131
File Output Format Counters
Bytes Written=9717
查看结果，执行如下命令：
hadoop fs -cat /output/wordcount/part-r-00000 | head
结果数据示例如下：
$ hadoop fs -cat /output/wordcount/part-r-00000 | head
13/12/25 22:58:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
"*" 17
"AS 3
"License"); 3
"alice,bob 17
$HADOOP_HOME/share/hadoop/common/lib/*, 1
$HADOOP_HOME/share/hadoop/hdfs/*,$HADOOP_HOME/share/hadoop/hdfs/lib/*, 1
$HADOOP_HOME/share/hadoop/mapreduce/*,$HADOOP_HOME/share/hadoop/mapreduce/lib/*</value> 1
$HADOOP_HOME/share/hadoop/yarn/*,$HADOOP_HOME/share/hadoop/yarn/lib/*, 1
(ASF) 1
(YARN-1229)--> 1
cat: Unable to write to output stream.
登录到Web控制台，访问链接http://m1:8088/可以看到任务记录情况。
可见，我们的HDFS能够存储数据，而YARN集群也能够运行MapReduce任务。
问题及总结需要知道的默认配置在Hadoop 2.2.0中，YARN框架有很多默认的参数值，如果你是在机器资源比较不足的情况下，需要修改这些默认值，来满足一些任务需要。NodeManager和ResourceManager都是在yarn-site.xml文件中配置的，而运行MapReduce任务时，是在mapred-site.xml中进行配置的。
下面看一下相关的参数及其默认值情况：
参数名称默认值进程名称配置文件含义说明
yarn.nodemanager.resource.memory-mb8192NodeManageryarn-site.xml从节点所在物理主机的可用物理内存总量
yarn.nodemanager.resource.cpu-vcores8NodeManageryarn-site.xml节点所在物理主机的可用虚拟CPU资源总数（core）
yarn.nodemanager.vmem-pmem-ratio2.1NodeManageryarn-site.xml使用1M物理内存，最多可以使用的虚拟内存数量
yarn.scheduler.minimum-allocation-mb1024ResourceManageryarn-site.xml一次申请分配内存资源的最小数量
yarn.scheduler.maximum-allocation-mb8192ResourceManageryarn-site.xml一次申请分配内存资源的最大数量
yarn.scheduler.minimum-allocation-vcores1ResourceManageryarn-site.xml一次申请分配虚拟CPU资源最小数量
yarn.scheduler.maximum-allocation-vcores8ResourceManageryarn-site.xml一次申请分配虚拟CPU资源最大数量
mapreduce.framework.namelocalMapReducemapred-site.xml取值local、classic或yarn其中之一，如果不是yarn，则不会使用YARN集群来实现资源的分配
mapreduce.map.memory.mb1024MapReducemapred-site.xml每个MapReduce作业的map任务可以申请的内存资源数量
mapreduce.map.cpu.vcores1MapReducemapred-site.xml每个MapReduce作业的map任务可以申请的虚拟CPU资源的数量
mapreduce.reduce.memory.mb1024MapReducemapred-site.xml每个MapReduce作业的reduce任务可以申请的内存资源数量
yarn.nodemanager.resource.cpu-vcores8MapReducemapred-site.xml每个MapReduce作业的reduce任务可以申请的虚拟CPU资源的数量
异常java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.66:50010详细异常信息，如下所示：$ hadoop fs -put /home/shirdrn/cloud/programs/hadoop-2.2.0/etc/hadoop/*.xml /data/wordcount/
13/12/25 21:29:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.66:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1166)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-1388035628061:blk_1073741825_1001
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.66:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 10.95.3.59:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1166)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-1388035628061:blk_1073741826_1002
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.59:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1128)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
13/12/25 21:29:46 INFO hdfs.DFSClient: Abandoning BP-1906424073-10.95.3.48-
1388035628061:blk_1073741828_1004
13/12/25 21:29:46 INFO hdfs.DFSClient: Excluding datanode 10.95.3.59:50010
13/12/25 21:29:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
主要是由于Hadoop集群内某些节点的防火墙没有关闭，导致无法访问集群内节点。
如果你的能力还不能撑起你的欲望，请静下心来学习！！
本帖，完毕。
如果还有问题可查看：
hadoop2.X使用手册1：通过web端口查看主节点、slave1节点及集群运行状态
hadoop2.X使用手册2：如何运行自带wordcount
hadoop2.2运行mapreduce（wordcount）问题总结
分享自：简单之美
页:
[1]