hadoop2.5.2汇总:新增功能最新编译 64位安装、源码包、API、eclipse插件下载
1.hadoop2.5.2发布:新增功能和改进
hadoop2.5.2与hadoop2.5.1新增功能和改进几乎没有变化
Common:
1.当使用代理服务器访问WebHDFS的时候,使用的 HTTP proxy server的认证有所改进。
2.使用新的 metrics sink,它允许直接写Graphite
3.规范 Hadoop兼容文件系统(HCFS) 相关工作
HDFS:
1.支持POSIX风格的文件系统的扩展属性,详细查看用户文档
2.使用OfflineImageViewer,客户端可以通过WebHDFS API浏览fsimage
3.NFS网关做了不少的改进和修复的bug,Hadoop portmapper 不再需要运行gateway,gateway 能够拒绝连接从非特权的端口中
4.SecondaryNameNode, JournalNode, 和DataNode web界面使用了HTML5 和Javascript技术.
YARN
1.YARN's REST APIs现在支持 写/修改 操作,用户可以通过 REST APIs提交和杀死应用程序
2.在YARN的timeline存储,用于存储通用的和指定的应用程序信息,支持通过Kerberos认证
3.公平调度器支持动态分级用户队列,队列是在运行时任何指定的父队列下动态创建的
2.相关包下载:
hadoop-aboutyun-linux64-2.5.2.tar.gz
链接:http://pan.baidu.com/s/1bnH94xp 密码:3fp5
hadoop-eclipse-plugin-2.5.2.jar
链接:http://pan.baidu.com/s/1pJx8llx 密码:**** Hidden Message *****
eclipse jee luna R win32 x86 64
Developers Version: Luna Release 4 4 0 Build id: 20140612 0600
hadoop2.5.2官网下载
hadoop-2.5.2-官网下载.tar.gz(下载后,名字可自行修改)
链接:http://pan.baidu.com/s/1eQpdf1K 密码:slzt
hadoop-2.5.2-src.tar(此为maven包)
链接:http://pan.baidu.com/s/1o6HneZ4 密码:m62w
hadoop-2.5.2源码获取参考
从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码
hadoop-2.5.2-api(此为文件)
链接:http://pan.baidu.com/s/1ntuo2Xr 密码:qzex
不知如何查看,可以参考如何找到hadoop2.x的api文档
static/image/hrline/2.gif
编译hadoop2.5.2经验总结
编译的时候遇到了不少问题,在后面会贴出错误,参考
从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4环境重新安装了一遍,但是却没有预想的那么顺利,遇到了不少问题,最终归结于编译环境不纯,什么是不纯那,举个例子
当时想通过在线安装maven,可是网络实在是太慢了,所以终止在线安装,直接从网上了下载了个离线包,然后配置了一下,结果编译不成功。
最后产生了如下的问题:
问题1:
------------------------------------------------------------------------
Reactor Summary:
Apache Hadoop Main ................................. SUCCESS
Apache Hadoop Project POM .......................... FAILURE
Apache Hadoop Annotations .......................... SKIPPED
Apache Hadoop Assemblies ........................... SKIPPED
Apache Hadoop Project Dist POM ..................... SKIPPED
Apache Hadoop Maven Plugins ........................ SKIPPED
Apache Hadoop MiniKDC .............................. SKIPPED
Apache Hadoop Auth ................................. SKIPPED
Apache Hadoop Auth Examples ........................ SKIPPED
Apache Hadoop Common ............................... SKIPPED
Apache Hadoop NFS .................................. SKIPPED
Apache Hadoop Common Project ....................... SKIPPED
Apache Hadoop HDFS ................................. SKIPPED
Apache Hadoop HttpFS ............................... SKIPPED
Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
Apache Hadoop HDFS-NFS ............................. SKIPPED
Apache Hadoop HDFS Project ......................... SKIPPED
hadoop-yarn ........................................ SKIPPED
hadoop-yarn-api .................................... SKIPPED
hadoop-yarn-common ................................. SKIPPED
hadoop-yarn-server ................................. SKIPPED
hadoop-yarn-server-common .......................... SKIPPED
hadoop-yarn-server-nodemanager ..................... SKIPPED
hadoop-yarn-server-web-proxy ....................... SKIPPED
hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
hadoop-yarn-server-resourcemanager ................. SKIPPED
hadoop-yarn-server-tests ........................... SKIPPED
hadoop-yarn-client ................................. SKIPPED
hadoop-yarn-applications ........................... SKIPPED
hadoop-yarn-applications-distributedshell .......... SKIPPED
hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
hadoop-yarn-site ................................... SKIPPED
hadoop-yarn-project ................................ SKIPPED
hadoop-mapreduce-client ............................ SKIPPED
hadoop-mapreduce-client-core ....................... SKIPPED
hadoop-mapreduce-client-common ..................... SKIPPED
hadoop-mapreduce-client-shuffle .................... SKIPPED
hadoop-mapreduce-client-app ........................ SKIPPED
hadoop-mapreduce-client-hs ......................... SKIPPED
hadoop-mapreduce-client-jobclient .................. SKIPPED
hadoop-mapreduce-client-hs-plugins ................. SKIPPED
Apache Hadoop MapReduce Examples ................... SKIPPED
hadoop-mapreduce ................................... SKIPPED
Apache Hadoop MapReduce Streaming .................. SKIPPED
Apache Hadoop Distributed Copy ..................... SKIPPED
Apache Hadoop Archives ............................. SKIPPED
Apache Hadoop Rumen ................................ SKIPPED
Apache Hadoop Gridmix .............................. SKIPPED
Apache Hadoop Data Join ............................ SKIPPED
Apache Hadoop Extras ............................... SKIPPED
Apache Hadoop Pipes ................................ SKIPPED
Apache Hadoop OpenStack support .................... SKIPPED
Apache Hadoop Client ............................... SKIPPED
Apache Hadoop Mini-Cluster ......................... SKIPPED
Apache Hadoop Scheduler Load Simulator ............. SKIPPED
Apache Hadoop Tools Dist ........................... SKIPPED
Apache Hadoop Tools ................................ SKIPPED
Apache Hadoop Distribution ......................... SKIPPED
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time: 40:20 min
Finished at: 2014-11-23T00:36:03-08:00
Final Memory: 32M/84M
------------------------------------------------------------------------
Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-javadoc-plugin:jar:2.8.1 -> org.apache.maven.shared:maven-common-artifact-filters:jar:1.3: Failed to read artifact descriptor for org.apache.maven.shared:maven-common-artifact-filters:jar:1.3: Could not transfer artifact org.apache.maven:maven-parent:pom:13 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/maven/maven-parent/13/maven-parent-13.pom from central failed: Read timed out ->
问题2
Apache Hadoop Main ................................. SUCCESS
Apache Hadoop Project POM .......................... SUCCESS
Apache Hadoop Annotations .......................... SUCCESS
Apache Hadoop Assemblies ........................... SUCCESS
Apache Hadoop Project Dist POM ..................... SUCCESS
Apache Hadoop Maven Plugins ........................ SUCCESS
Apache Hadoop MiniKDC .............................. FAILURE
Apache Hadoop Auth ................................. SKIPPED
Apache Hadoop Auth Examples ........................ SKIPPED
Apache Hadoop Common ............................... SKIPPED
Apache Hadoop NFS .................................. SKIPPED
Apache Hadoop Common Project ....................... SKIPPED
Apache Hadoop HDFS ................................. SKIPPED
Apache Hadoop HttpFS ............................... SKIPPED
Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
Apache Hadoop HDFS-NFS ............................. SKIPPED
Apache Hadoop HDFS Project ......................... SKIPPED
hadoop-yarn ........................................ SKIPPED
hadoop-yarn-api .................................... SKIPPED
hadoop-yarn-common ................................. SKIPPED
hadoop-yarn-server ................................. SKIPPED
hadoop-yarn-server-common .......................... SKIPPED
hadoop-yarn-server-nodemanager ..................... SKIPPED
hadoop-yarn-server-web-proxy ....................... SKIPPED
hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
hadoop-yarn-server-resourcemanager ................. SKIPPED
hadoop-yarn-server-tests ........................... SKIPPED
hadoop-yarn-client ................................. SKIPPED
hadoop-yarn-applications ........................... SKIPPED
hadoop-yarn-applications-distributedshell .......... SKIPPED
hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
hadoop-yarn-site ................................... SKIPPED
hadoop-yarn-project ................................ SKIPPED
hadoop-mapreduce-client ............................ SKIPPED
hadoop-mapreduce-client-core ....................... SKIPPED
hadoop-mapreduce-client-common ..................... SKIPPED
hadoop-mapreduce-client-shuffle .................... SKIPPED
hadoop-mapreduce-client-app ........................ SKIPPED
hadoop-mapreduce-client-hs ......................... SKIPPED
hadoop-mapreduce-client-jobclient .................. SKIPPED
hadoop-mapreduce-client-hs-plugins ................. SKIPPED
Apache Hadoop MapReduce Examples ................... SKIPPED
hadoop-mapreduce ................................... SKIPPED
Apache Hadoop MapReduce Streaming .................. SKIPPED
Apache Hadoop Distributed Copy ..................... SKIPPED
Apache Hadoop Archives ............................. SKIPPED
Apache Hadoop Rumen ................................ SKIPPED
Apache Hadoop Gridmix .............................. SKIPPED
Apache Hadoop Data Join ............................ SKIPPED
Apache Hadoop Extras ............................... SKIPPED
Apache Hadoop Pipes ................................ SKIPPED
Apache Hadoop OpenStack support .................... SKIPPED
Apache Hadoop Client ............................... SKIPPED
Apache Hadoop Mini-Cluster ......................... SKIPPED
Apache Hadoop Scheduler Load Simulator ............. SKIPPED
Apache Hadoop Tools Dist ........................... SKIPPED
Apache Hadoop Tools ................................ SKIPPED
Apache Hadoop Distribution ......................... SKIPPED
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time: 22.376 s
Finished at: 2014-11-23T04:20:31-08:00
Final Memory: 38M/91M
------------------------------------------------------------------------
Failed to execute goal on project hadoop-minikdc: Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.5.2: Failure to find org.apache.directory.server:apacheds-core:jar:2.0.0-M15 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced ->
To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.
For more information about the errors and possible solutions, please read the following articles:
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
After correcting the problems, you can resume the build with the command
mvn <goals> -rf :hadoop-minikdc
问题3
------------------------------------------------------------------------
Reactor Summary:
Apache Hadoop Main ................................. SUCCESS
Apache Hadoop Project POM .......................... SUCCESS
Apache Hadoop Annotations .......................... SUCCESS
Apache Hadoop Assemblies ........................... SUCCESS
Apache Hadoop Project Dist POM ..................... SUCCESS
Apache Hadoop Maven Plugins ........................ SUCCESS
Apache Hadoop Auth ................................. FAILURE
Apache Hadoop Auth Examples ........................ SKIPPED
Apache Hadoop Common ............................... SKIPPED
Apache Hadoop NFS .................................. SKIPPED
Apache Hadoop MiniKDC .............................. SKIPPED
Apache Hadoop Common Project ....................... SKIPPED
Apache Hadoop HDFS ................................. SKIPPED
Apache Hadoop HttpFS ............................... SKIPPED
Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
Apache Hadoop HDFS-NFS ............................. SKIPPED
Apache Hadoop HDFS Project ......................... SKIPPED
hadoop-yarn ........................................ SKIPPED
hadoop-yarn-api .................................... SKIPPED
hadoop-yarn-common ................................. SKIPPED
hadoop-yarn-server ................................. SKIPPED
hadoop-yarn-server-common .......................... SKIPPED
hadoop-yarn-server-nodemanager ..................... SKIPPED
hadoop-yarn-server-web-proxy ....................... SKIPPED
hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
hadoop-yarn-server-resourcemanager ................. SKIPPED
hadoop-yarn-server-tests ........................... SKIPPED
hadoop-yarn-client ................................. SKIPPED
hadoop-yarn-applications ........................... SKIPPED
hadoop-yarn-applications-distributedshell .......... SKIPPED
hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
hadoop-yarn-site ................................... SKIPPED
hadoop-yarn-project ................................ SKIPPED
hadoop-mapreduce-client ............................ SKIPPED
hadoop-mapreduce-client-core ....................... SKIPPED
hadoop-mapreduce-client-common ..................... SKIPPED
hadoop-mapreduce-client-shuffle .................... SKIPPED
hadoop-mapreduce-client-app ........................ SKIPPED
hadoop-mapreduce-client-hs ......................... SKIPPED
hadoop-mapreduce-client-jobclient .................. SKIPPED
hadoop-mapreduce-client-hs-plugins ................. SKIPPED
Apache Hadoop MapReduce Examples ................... SKIPPED
hadoop-mapreduce ................................... SKIPPED
Apache Hadoop MapReduce Streaming .................. SKIPPED
Apache Hadoop Distributed Copy ..................... SKIPPED
Apache Hadoop Archives ............................. SKIPPED
Apache Hadoop Rumen ................................ SKIPPED
Apache Hadoop Gridmix .............................. SKIPPED
Apache Hadoop Data Join ............................ SKIPPED
Apache Hadoop Extras ............................... SKIPPED
Apache Hadoop Pipes ................................ SKIPPED
Apache Hadoop OpenStack support .................... SKIPPED
Apache Hadoop Client ............................... SKIPPED
Apache Hadoop Mini-Cluster ......................... SKIPPED
Apache Hadoop Scheduler Load Simulator ............. SKIPPED
Apache Hadoop Tools Dist ........................... SKIPPED
Apache Hadoop Tools ................................ SKIPPED
Apache Hadoop Distribution ......................... SKIPPED
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time: 01:32 min
Finished at: 2014-11-23T04:50:17-08:00
Final Memory: 42M/102M
------------------------------------------------------------------------
Failed to execute goal on project hadoop-auth: Could not resolve dependencies for project org.apache.hadoop:hadoop-auth:jar:2.5.2: Failure to find org.apache.directory.server:apacheds-core:jar:2.0.0-M15 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced ->
To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.
For more information about the errors and possible solutions, please read the following articles:
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
After correcting the problems, you can resume the build with the command
mvn <goals> -rf :hadoop-auth
上面上的含义都是找不到包,可能的原因是找不到包,
但是我的网络又是通的,所以所以肯定是maven出了问题。
最后重新部署了干净的环境。
终于编译成功
http://www.aboutyun.com/data/attachment/forum/201411/24/004753tuppnpznz47yx4n4.png
http://www.aboutyun.com/data/attachment/forum/201411/24/004751fiacie9c999xgoxp.png
http://www.aboutyun.com/data/attachment/forum/201411/24/004835p7hvk5q43qr7e3dh.png
参考文档:
hadoop2.5.1汇总:最新编译 32位、64位安装、源码包、API下载及新特性等
从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4
相关内容:
hadoop2.6.0汇总:新增功能最新编译 32位、64位安装、源码包、API下载及部署文档
hadoop2.5.2汇总:新增功能最新编译 32位、64位安装、源码包、API、eclipse插件下载
hadoop2.5.1汇总:最新编译 32位、64位安装、源码包、API下载及新特性等
hadoop2.5.1汇总:最新编译 32位、64位安装、源码包、API下载及新特性等
新手指导:hadoop官网介绍及如何下载hadoop(2.4)各个版本与查看hadoop API介绍
从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4
谢谢楼主了,看看哈哈
楼主V5,收藏
楼主V587,分享了 赞啊{:soso_e130:} 好 ,不知道是不是真64啊
太好了,多谢啦! 感谢楼主分享……
谢谢楼主。。。。