windows通过hadoop-eclipse-plugin插件远程开发hadoop运行mapreduce遇到问题及解决

fzu3144 发表于 2015-7-15 16:29
hadoop 2.7版本，linux 6.6，
win7下用IDE跑MR出错。。。。求助~

你的mapreduce版本似乎是1.x的，引用的库确实2.7的。应该是版本兼容问题

我的hadoop2.6 centos64 位报这样的错

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NullPointerException
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at Mywordcount.main(Mywordcount.java:65)

栎梓天冲 发表于 2015-7-19 15:58
我的hadoop2.6 centos64 位报这样的错

log4j:WARN No appenders could be found for logger (org.apache ...

是在win7下吗？
可能缺少
(org.apache.hadoop.util.Shell).

如果是在win7下，建议下载这个插件放到相应的位置
hadoop2.6 window下eclipse开发hadoop需要添加的组件
http://www.aboutyun.com/thread-14155-1-1.html

这是怎么回事？？？
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
        at MyMapre.WordCount.main(WordCount.java:61)

栎梓天冲 发表于 2015-7-22 21:33
这是怎么回事？？？
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
...

少导入hadoop-hdfs.jar这个jar包

如不明确包的作用，建议全部倒入。

这是不是我伪分布测试出错的原因
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.util.Apps
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 24 more

栎梓天冲 发表于 2015-7-22 21:57
这是不是我伪分布测试出错的原因
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn. ...

还是缺少包

mark一下，我也在win下操作遇到了一些问题！

mark，学习

学习学习