运行wordcount报错

hyj 发表于 2017-3-15 11:07
贴出问题来

还是之前的第二个问题，貌似没有很好识别后面参数个数。

用这样的方式试试：
hadoop jar ~/workspace/HadoopTest02.jar wordcount.WordCount /user/hadoop/test1/input/*.txt  /user/hadoop/test1/output
这里需要指定你input具体的文件在哪里，因为你的input里面还有文件夹！

w517424787 发表于 2017-3-15 14:33
用这样的方式试试：
hadoop jar ~/workspace/HadoopTest02.jar wordcount.WordCount /user/hadoop/test1/i ...

input下面的四个是文件不是文件夹，是拿在外部vi直接创建的，然后put进来的。后缀没有显示出来。

丹青穆怀 发表于 2017-3-15 14:43
input下面的四个是文件不是文件夹，是拿在外部vi直接创建的，然后put进来的。后缀没有显示出来。

  if (otherArgs.length != 2) {
                        System.err.println("Usage:wordcount<in><out>");
                        System.exit(2);
                }
把上面代码改成下面
  if (otherArgs.length== 0) {
                        System.println(otherArgs[0]);
                        System.exit(2);
                }
  if (otherArgs.length == 1) {
                    System.println(otherArgs[0]);
                    System.println(otherArgs[1]);
                        System.exit(2);
                }
  if (otherArgs.length == 2) {
                     System.println(otherArgs[0]);
                    System.println(otherArgs[1]);
                    System.println(otherArgs[2]);
                        System.exit(2);
                }
  if (otherArgs.length == 3) {
                   System.println(otherArgs[0]);
                    System.println(otherArgs[1]);
                   System.println(otherArgs[3]);
                        System.exit(2);
                }
  if (otherArgs.length == 4) {
                 System.println(otherArgs[0]);
                    System.println(otherArgs[1]);
                   System.println(otherArgs[3]);
                        System.exit(2);
                }
看看都输出什么

楼主你不妨先在Hadoop自带例子中运行wordcount，看看是否是格式问题，如：
[mw_shl_code=bash,false]hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /data/input /data/output
[/mw_shl_code]

nextuser 发表于 2017-3-15 17:19
if (otherArgs.length != 2) {
                        System.err.println("Usage:wordcount");
   ...

改了之后，成了这样了hadoop jar ~/workspace/HadoopTest04.jar wordcount.WordCount /user/hadoop/test1/input /user/hadoop/test1/output
14/07/01 10:44:50 INFO impl.TimelineClientImpl: Timeline service address: http://hadoop001:8188/ws/v1/timeline/
14/07/01 10:44:51 INFO client.AHSProxy: Connecting to Application History server at hadoop001/172.16.13.224:10200
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://beh/user/hadoop/test1/input already exists
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:267)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:140)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1297)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1294)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1315)
        at wordcount.WordCount.main(WordCount.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

yongjian3311 发表于 2017-3-15 17:37
楼主你不妨先在Hadoop自带例子中运行wordcount，看看是否是格式问题，如：
[mw_shl_code=bash,false]hadoo ...

自带的例子也报错
hadoop jar /hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /user/hadoop/test1/input /user/hadoop/test1/output
14/07/01 10:49:50 INFO impl.TimelineClientImpl: Timeline service address: http://hadoop001:8188/ws/v1/timeline/
14/07/01 10:49:50 INFO client.AHSProxy: Connecting to Application History server at hadoop001/172.16.13.224:10200
14/07/01 10:49:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got access token error for connect ack with firstBadLink as 172.16.13.226:50010
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1456)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
14/07/01 10:49:51 INFO hdfs.DFSClient: Abandoning BP-516167770-172.16.13.224-1488772634200:blk_1073742468_1644
14/07/01 10:49:51 INFO hdfs.DFSClient: Excluding datanode 172.16.13.226:50010
14/07/01 10:49:51 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /opt/beh/data/yarn/staging/hadoop/.staging/job_1404180478524_0002/job.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)

        at org.apache.hadoop.ipc.Client.call(Client.java:1469)
        at org.apache.hadoop.ipc.Client.call(Client.java:1400)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
14/07/01 10:49:51 INFO mapreduce.JobSubmitter: Cleaning up the staging area /opt/beh/data/yarn/staging/hadoop/.staging/job_1404180478524_0002
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /opt/beh/data/yarn/staging/hadoop/.staging/job_1404180478524_0002/job.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)

        at org.apache.hadoop.ipc.Client.call(Client.java:1469)
        at org.apache.hadoop.ipc.Client.call(Client.java:1400)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)

丹青穆怀 发表于 2017-3-16 11:04
自带的例子也报错
hadoop jar /hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar w ...

jps看下进程是否还在。应该是集群的问题。

easthome001 发表于 2017-3-16 16:27
jps看下进程是否还在。应该是集群的问题。

进程都在，现在是不知道集群里面哪个配置出问题了。

丹青穆怀 发表于 2017-3-17 13:52
进程都在，现在是不知道集群里面哪个配置出问题了。

File /opt/beh/data/yarn/staging/hadoop/.staging/job_1404180478524_0002/job.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
这里明显你的datanode坏了一个，重启下试试，可能有僵尸进程，并且都等5分钟后再看，有的起来后就挂掉了。每台都看下。最好贴出来，别人帮你看下。