有关wordcount的问题,无法查看输出目录
我的输入目录是/input,输出目录为/tmp/output,运行时输出如下:12/08/19 17:23:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/08/19 17:23:14 WARN mapred.JobClient: No job jar file set.User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****file:/input
12/08/19 17:23:14 INFO input.FileInputFormat: Total input paths to process : 1
12/08/19 17:23:14 WARN snappy.LoadSnappy: Snappy native library not loaded
12/08/19 17:23:14 INFO mapred.JobClient: Running job: job_local_0001
12/08/19 17:23:14 INFO mapred.Task:Using ResourceCalculatorPlugin : null
12/08/19 17:23:14 INFO mapred.MapTask: io.sort.mb = 100
12/08/19 17:23:15 INFO mapred.JobClient:map 0% reduce 0%
12/08/19 17:23:17 INFO mapred.MapTask: data buffer = 79691776/99614720
12/08/19 17:23:17 INFO mapred.MapTask: record buffer = 262144/327680
12/08/19 17:23:17 INFO mapred.MapTask: Starting flush of map output
12/08/19 17:23:17 INFO mapred.MapTask: Finished spill 0
12/08/19 17:23:17 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/08/19 17:23:17 INFO mapred.LocalJobRunner:
12/08/19 17:23:17 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
12/08/19 17:23:17 INFO mapred.Task:Using ResourceCalculatorPlugin : null
12/08/19 17:23:17 INFO mapred.LocalJobRunner:
12/08/19 17:23:17 INFO mapred.Merger: Merging 1 sorted segments
12/08/19 17:23:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 89 bytes
12/08/19 17:23:17 INFO mapred.LocalJobRunner:
12/08/19 17:23:17 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/08/19 17:23:17 INFO mapred.LocalJobRunner:
12/08/19 17:23:17 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/08/19 17:23:17 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /tmp/output
12/08/19 17:23:17 INFO mapred.JobClient:map 100% reduce 0%
12/08/19 17:23:20 INFO mapred.LocalJobRunner: reduce > reduce
12/08/19 17:23:20 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/08/19 17:23:20 INFO mapred.JobClient:map 100% reduce 100%
12/08/19 17:23:20 INFO mapred.JobClient: Job complete: job_local_0001
12/08/19 17:23:20 INFO mapred.JobClient: Counters: 17
12/08/19 17:23:20 INFO mapred.JobClient: File Output Format Counters
12/08/19 17:23:20 INFO mapred.JobClient: Bytes Written=95
12/08/19 17:23:20 INFO mapred.JobClient: FileSystemCounters
12/08/19 17:23:20 INFO mapred.JobClient: FILE_BYTES_READ=561
12/08/19 17:23:20 INFO mapred.JobClient: FILE_BYTES_WRITTEN=65103
12/08/19 17:23:20 INFO mapred.JobClient: File Input Format Counters
12/08/19 17:23:20 INFO mapred.JobClient: Bytes Read=92
12/08/19 17:23:20 INFO mapred.JobClient: Map-Reduce Framework
12/08/19 17:23:20 INFO mapred.JobClient: Map output materialized bytes=93
12/08/19 17:23:20 INFO mapred.JobClient: Map input records=1
12/08/19 17:23:20 INFO mapred.JobClient: Reduce shuffle bytes=0
12/08/19 17:23:20 INFO mapred.JobClient: Spilled Records=2
12/08/19 17:23:20 INFO mapred.JobClient: Map output bytes=85
12/08/19 17:23:20 INFO mapred.JobClient: Total committed heap usage (bytes)=365404160
12/08/19 17:23:20 INFO mapred.JobClient: SPLIT_RAW_BYTES=88
12/08/19 17:23:20 INFO mapred.JobClient: Combine input records=1
12/08/19 17:23:20 INFO mapred.JobClient: Reduce input records=1
12/08/19 17:23:20 INFO mapred.JobClient: Reduce input groups=1
12/08/19 17:23:20 INFO mapred.JobClient: Combine output records=1
12/08/19 17:23:21 INFO mapred.JobClient: Reduce output records=1
12/08/19 17:23:21 INFO mapred.JobClient: Map output records=1
我是用wordcount源代码在eclipse里面运行的,然后在cygwin中看不到/tmp/output目录,
我以为是程序没有生成该目录,然后将代码又跑了一遍,提示如下:
12/08/19 17:32:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/08/19 17:32:52 WARN mapred.JobClient: No job jar file set.User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/08/19 17:32:52 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-Administrator/mapred/staging/Administrator-1886074235/.staging/job_local_0001
12/08/19 17:32:52 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/output already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/output already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:889)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at com.style.hadoop.WordCount.main(WordCount.java:67)
提示/tmp/output目录已存在,可是在cygwin中用命令hadoop fs -lsr / 查看不到该目录,并且还能新建/tmp/output目录,求解答,我想是不是因为用户,本人是初学者,不知如何解决,请求高手帮助,谢谢! 真的,cygwin模拟会出很多,hadoop就是为linux设计的系统,真心希望小学弟们不要再win下测试了。很多都不是自己的,都是这个模拟器不够给力。
要真的想学习还是要用linux系统,没有这个条件,就启动虚拟机装一个linux的测试。
页:
[1]