hadoop2.2的环境问题

本帖最后由 zhangcd123 于 2014-4-28 17:42 编辑

运行hadoop2.2的样例MapReduce程序时出现如下问题exitCode:126 due to:Exception from container-launch:

sstutu · 发表于 2014-4-28 17:50:10

本帖最后由 sstutu 于 2014-4-28 17:55 编辑

你把详细信息贴出来，包括运行的命令，报错的日志.是在集群上运行，还是在eclipse运行

zhangcd123 · 发表于 2014-4-28 18:11:33

执行命令如下：
hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-102.jar wordcount /input /output

错误信息如下：
14/04/28 18:10:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/28 18:10:41 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.250:9010
14/04/28 18:10:43 INFO input.FileInputFormat: Total input paths to process : 0
14/04/28 18:10:43 INFO mapreduce.JobSubmitter: number of splits:0
14/04/28 18:10:43 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/04/28 18:10:43 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/04/28 18:10:43 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/04/28 18:10:43 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/04/28 18:10:43 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/04/28 18:10:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1398679196466_0003
14/04/28 18:10:44 INFO impl.YarnClientImpl: Submitted application application_1398679196466_0003 to ResourceManager at /192.168.1.250:9010
14/04/28 18:10:44 INFO mapreduce.Job: The url to track the job: http://http://hadoop:9001/proxy/application_1398679196466_0003/
14/04/28 18:10:44 INFO mapreduce.Job: Running job: job_1398679196466_0003
14/04/28 18:10:48 INFO mapreduce.Job: Job job_1398679196466_0003 running in uber mode : false
14/04/28 18:10:48 INFO mapreduce.Job:  map 0% reduce 0%
14/04/28 18:10:48 INFO mapreduce.Job: Job job_1398679196466_0003 failed with state FAILED due to: Application application_1398679196466_0003 failed 2 times due to AM Container for appattempt_1398679196466_0003_000002 exited with  exitCode: 126 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
      at org.apache.hadoop.util.Shell.run(Shell.java:379)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
      at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      at java.lang.Thread.run(Thread.java:662)

.Failing this attempt.. Failing the application.
14/04/28 18:10:48 INFO mapreduce.Job: Counters: 0

pig2 · 发表于 2014-4-28 18:30:38

本帖最后由 pig2 于 2014-4-28 18:32 编辑

zhangcd123 发表于 2014-4-28 18:11
执行命令如下：
hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6. ...

你的配置端口是不是按照hadoop1.x来的hadoop2你可以参考下面的端口：(你的9010改成8032试一下)

hadoop2的更多端口，你可以看看hadoop2.x常用端口及定义方法（可收藏方便以后查询）

zhangcd123 · 发表于 2014-4-28 18:38:59

@pig 2 你好，那个端口改成8032也不行

pig2 · 发表于 2014-4-28 18:50:35

本帖最后由 pig2 于 2014-4-28 18:54 编辑

zhangcd123 发表于 2014-4-28 18:38
@pig 2 你好，那个端口改成8032也不行

不好定位可能错误的原因：
1.检查一下hadoop API是使用旧的还是新的
2.检查一下你的输入输出路径（特别是输入路径，你的输入文件是怎么指定的）
3.你的配置文件，是否其他地方还没有修改。
你逐一排查一下。不确定的就贴出来。

zhangcd123 · 发表于 2014-4-28 18:53:36

1、运行的是hadoop的例子程序应该跟hadoopAPI无关
2、输入路径存在
3、只做了一个节点所以配置文件也改了

pig2 · 发表于 2014-4-28 18:57:15

zhangcd123 发表于 2014-4-28 18:53
1、运行的是hadoop的例子程序应该跟hadoopAPI无关
2、输入路径存在
3、只做了一个节点所以配置文件也改了

你是伪分布？你是按照什么文档配置的，还有你的配置文件，只有输入路径可能不行，需要找到输入的文件，都贴出来看一下。

zhangcd123 · 发表于 2014-4-28 19:01:13

core-site.xml:
<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.1.250:9000</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/hadoop/tmp</value>
</property>
</configuration>

hdfs-site.xml:
<configuration>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/usr/hadoop/hdfs/name</value>
  <final>true</final>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/usr/hadoop/hdfs/data</value>
  <final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>

mapred-site.xml
<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>192.168.1.250:19888</value>
</property>
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>1536</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1440M</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>3072</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx2560M</value>
</property>
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx4096m</value>
</property>
<property>
  <name>mapreduce.task.io.sort.mb</name>
  <value>512</value>
</property>
<property>
  <name>mapreduce.task.io.sort.factor</name>
  <value>100</value>
</property>
<property>
  <name>mapreduce.reduce.shuffle.parallelcopies</name>
  <value>50</value>
</property>
<property>
  <name>mapred.job.resuse.jvm.num.tasks</name>
  <value>50</value>
</property>
</configuration>

yarn-site.xml:
<configuration>
<property>
  <name>yarn.resourcemanager.address</name>
  <value>192.168.1.250:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>192.168.1.250:9011</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.1.250:9001</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.1.250:9012</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>192.168.1.250:9020</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>2048</value>
</property>
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>9000</value>
</property>
<property>
  <name>yarn.nodemanager.resource.cpu-cores</name>
  <value>8</value>
</property>
</configuration>

输入文件为空，为空的时候wordcount不是也可以运行吗