图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Hadoop|YARN › 实操演练› Hadoop2.2.0 实例测试 WordCount程序
分享

Hadoop2.2.0 实例测试 WordCount程序

xioaxu790 2014-5-28 12:50:03 发表于 实操演练 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 1 7644
问题导读：
1、部署好Hadoop之后，如何运行一个优美的WordCount ？
2、我们需要做哪些准备 ？





前言
装好的hadoop测试一1个示例程序WordCount，首先需要在操作系统上新建两个任意文件，然后上传到hadoop，再运行该程序统计文件中单词的个数，最后查看结果。


在操作系统上新建任意文件：
例如：
[hadoop@hadoop01 input]$ ls
test1.txt  test2.txt

查看hadoop的文件系统目录：
  1.  [hadoop@hadoop01 input]$ hadoop fs -ls /

  2. Found 1 items

  3. drwxr-xr-x   - hadoop supergroup          0 2013-10-30 00:00 /input
复制代码

上传至hadoop的/input下：
  1. [hadoop@hadoop01 input]$ hadoop fs -put ./ /input

  2. [hadoop@hadoop01 input]$ hadoop fs -ls /input

  3. Found 2 items

  4. -rw-r--r--   3 hadoop supergroup         12 2013-10-30 00:00 /input/test1.txt

  5. -rw-r--r--   3 hadoop supergroup         13 2013-10-30 00:00 /input/test2.txt
复制代码

在hadoop文件系统命令查看这两个文件的内容：
  1.  [hadoop@hadoop01 test]$ hadoop fs -cat /input/test1.txt

  2. hello world

  3. [hadoop@hadoop01 test]$ hadoop fs -cat /input/test2.txt

  4. hello hadoop
复制代码

运行示例程序（WordCount)： 
  1. [hadoop@hadoop01 test]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.2.0-sources.jar org.apache.hadoop.examples.WordCount /input /output

  2. 13/11/06 21:33:40 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id

  3. 13/11/06 21:33:40 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

  4. 13/11/06 21:33:40 INFO input.FileInputFormat: Total input paths to process : 2

  5. 13/11/06 21:33:41 INFO mapreduce.JobSubmitter: number of splits:2

  6. 13/11/06 21:33:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name

  7. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar

  8. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class

  9. 13/11/06 21:33:41 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class

  10. 13/11/06 21:33:41 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

  11. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name

  12. 13/11/06 21:33:41 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

  13. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir

  14. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

  15. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

  16. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class

  17. 13/11/06 21:33:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

  18. 13/11/06 21:33:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local382050821_0001

  19. 13/11/06 21:33:41 WARN conf.Configuration: file:/hadoop/hdfs/tmp/hadoop-hadoop/mapred/staging/hadoop382050821/.staging/job_local382050821_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.

  20. 13/11/06 21:33:41 WARN conf.Configuration: file:/hadoop/hdfs/tmp/hadoop-hadoop/mapred/staging/hadoop382050821/.staging/job_local382050821_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

  21. 13/11/06 21:33:42 WARN conf.Configuration: file:/hadoop/hdfs/tmp/hadoop-hadoop/mapred/local/localRunner/hadoop/job_local382050821_0001/job_local382050821_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.

  22. 13/11/06 21:33:42 WARN conf.Configuration: file:/hadoop/hdfs/tmp/hadoop-hadoop/mapred/local/localRunner/hadoop/job_local382050821_0001/job_local382050821_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

  23. 13/11/06 21:33:42 INFO mapreduce.Job: The url to track the job: :8080/

  24. 13/11/06 21:33:42 INFO mapreduce.Job: Running job: job_local382050821_0001

  25. 13/11/06 21:33:42 INFO mapred.LocalJobRunner: OutputCommitter set in config null

  26. 13/11/06 21:33:42 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

  27. 13/11/06 21:33:42 INFO mapred.LocalJobRunner: Waiting for map tasks

  28. 13/11/06 21:33:42 INFO mapred.LocalJobRunner: Starting task: attempt_local382050821_0001_m_000000_0

  29. 13/11/06 21:33:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]

  30. 13/11/06 21:33:42 INFO mapred.MapTask: Processing split: hdfs://hadoop01:9000/input/test2.txt:0+13

  31. 13/11/06 21:33:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

  32. 13/11/06 21:33:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

  33. 13/11/06 21:33:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

  34. 13/11/06 21:33:42 INFO mapred.MapTask: soft limit at 83886080

  35. 13/11/06 21:33:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

  36. 13/11/06 21:33:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

  37. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: 

  38. 13/11/06 21:33:43 INFO mapred.MapTask: Starting flush of map output

  39. 13/11/06 21:33:43 INFO mapred.MapTask: Spilling map output

  40. 13/11/06 21:33:43 INFO mapred.MapTask: bufstart = 0; bufend = 21; bufvoid = 104857600

  41. 13/11/06 21:33:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600

  42. 13/11/06 21:33:43 INFO mapred.MapTask: Finished spill 0

  43. 13/11/06 21:33:43 INFO mapred.Task: Task:attempt_local382050821_0001_m_000000_0 is done. And is in the process of committing

  44. 13/11/06 21:33:43 INFO mapreduce.Job: Job job_local382050821_0001 running in uber mode : false

  45. 13/11/06 21:33:43 INFO mapreduce.Job:  map 0% reduce 0%

  46. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: map

  47. 13/11/06 21:33:43 INFO mapred.Task: Task 'attempt_local382050821_0001_m_000000_0' done.

  48. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local382050821_0001_m_000000_0

  49. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: Starting task: attempt_local382050821_0001_m_000001_0

  50. 13/11/06 21:33:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]

  51. 13/11/06 21:33:43 INFO mapred.MapTask: Processing split: hdfs://hadoop01:9000/input/test1.txt:0+12

  52. 13/11/06 21:33:43 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

  53. 13/11/06 21:33:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

  54. 13/11/06 21:33:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

  55. 13/11/06 21:33:43 INFO mapred.MapTask: soft limit at 83886080

  56. 13/11/06 21:33:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

  57. 13/11/06 21:33:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

  58. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: 

  59. 13/11/06 21:33:43 INFO mapred.MapTask: Starting flush of map output

  60. 13/11/06 21:33:43 INFO mapred.MapTask: Spilling map output

  61. 13/11/06 21:33:43 INFO mapred.MapTask: bufstart = 0; bufend = 20; bufvoid = 104857600

  62. 13/11/06 21:33:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600

  63. 13/11/06 21:33:43 INFO mapred.MapTask: Finished spill 0

  64. 13/11/06 21:33:43 INFO mapred.Task: Task:attempt_local382050821_0001_m_000001_0 is done. And is in the process of committing

  65. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: map

  66. 13/11/06 21:33:43 INFO mapred.Task: Task 'attempt_local382050821_0001_m_000001_0' done.

  67. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local382050821_0001_m_000001_0

  68. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: Map task executor complete.

  69. 13/11/06 21:33:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]

  70. 13/11/06 21:33:43 INFO mapred.Merger: Merging 2 sorted segments

  71. 13/11/06 21:33:43 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 36 bytes

  72. 13/11/06 21:33:43 INFO mapred.LocalJobRunner: 

  73. 13/11/06 21:33:43 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

  74. 13/11/06 21:33:44 INFO mapreduce.Job:  map 100% reduce 0%

  75. 13/11/06 21:33:44 INFO mapred.Task: Task:attempt_local382050821_0001_r_000000_0 is done. And is in the process of committing

  76. 13/11/06 21:33:44 INFO mapred.LocalJobRunner: 

  77. 13/11/06 21:33:44 INFO mapred.Task: Task attempt_local382050821_0001_r_000000_0 is allowed to commit now

  78. 13/11/06 21:33:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local382050821_0001_r_000000_0' to hdfs://hadoop01:9000/output/_temporary/0/task_local382050821_0001_r_000000

  79. 13/11/06 21:33:44 INFO mapred.LocalJobRunner: reduce > reduce

  80. 13/11/06 21:33:44 INFO mapred.Task: Task 'attempt_local382050821_0001_r_000000_0' done.

  81. 13/11/06 21:33:45 INFO mapreduce.Job:  map 100% reduce 100%

  82. 13/11/06 21:33:45 INFO mapreduce.Job: Job job_local382050821_0001 completed successfully

  83. 13/11/06 21:33:45 INFO mapreduce.Job: Counters: 32

  84.         File System Counters

  85.                 FILE: Number of bytes read=812174

  86.                 FILE: Number of bytes written=1395157

  87.                 FILE: Number of read operations=0

  88.                 FILE: Number of large read operations=0

  89.                 FILE: Number of write operations=0

  90.                 HDFS: Number of bytes read=63

  91.                 HDFS: Number of bytes written=25

  92.                 HDFS: Number of read operations=25

  93.                 HDFS: Number of large read operations=0

  94.                 HDFS: Number of write operations=5

  95.         Map-Reduce Framework

  96.                 Map input records=2

  97.                 Map output records=4

  98.                 Map output bytes=41

  99.                 Map output materialized bytes=61

  100.                 Input split bytes=202

  101.                 Combine input records=4

  102.                 Combine output records=4

  103.                 Reduce input groups=3

  104.                 Reduce shuffle bytes=0

  105.                 Reduce input records=4

  106.                 Reduce output records=3

  107.                 Spilled Records=8

  108.                 Shuffled Maps =0

  109.                 Failed Shuffles=0

  110.                 Merged Map outputs=0

  111.                 GC time elapsed (ms)=146

  112.                 CPU time spent (ms)=0

  113.                 Physical memory (bytes) snapshot=0

  114.                 Virtual memory (bytes) snapshot=0

  115.                 Total committed heap usage (bytes)=456732672

  116.         File Input Format Counters 

  117.                 Bytes Read=25

  118.         File Output Format Counters 

  119.                 Bytes Written=25
复制代码

查看程序运行结果：
  1.  [hadoop@hadoop01 test]$ hadoop fs -cat /output/part-r-00000

  2. hadoop  1

  3. hello   2

  4. world   1
复制代码




关注公众号，获取大数据、人工智能20套、区块链资源5阶段等资源，随时更新，获取最新技术资源
回复

使用道具 举报

自定义广告语

已有(1)人评论

跳转到指定楼层
正序浏览
2278 发表于 2014-7-18 17:44:00
学习了顶顶，。
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
xioaxu790

金牌会员

关注

  • 851

    主题

  • 1000

    帖子

  • 108

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000