图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Hadoop|YARN › 总结型› hadoop2.5.2汇总：新增功能最新编译 64位安装、源码包、 ...
分享

hadoop2.5.2汇总：新增功能最新编译 64位安装、源码包、API、eclipse插件下载

 
nettman 2014-11-24 00:35:26 发表于 总结型 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 240 197958


1.hadoop2.5.2发布：新增功能和改进

hadoop2.5.2与hadoop2.5.1新增功能和改进几乎没有变化

Common：
1.当使用代理服务器访问WebHDFS的时候，使用的 HTTP proxy server的认证有所改进。
2.使用新的 metrics sink，它允许直接写Graphite
3.规范 Hadoop兼容文件系统(HCFS) 相关工作

HDFS：
1.支持POSIX风格的文件系统的扩展属性，详细查看用户文档
2.使用OfflineImageViewer，客户端可以通过WebHDFS API浏览fsimage
3.NFS网关做了不少的改进和修复的bug,Hadoop portmapper 不再需要运行gateway，gateway 能够拒绝连接从非特权的端口中
4.SecondaryNameNode, JournalNode, 和DataNode web界面使用了HTML5 和Javascript技术.

YARN
1.YARN's REST APIs现在支持 写/修改 操作，用户可以通过 REST APIs提交和杀死应用程序
2.在YARN的timeline存储，用于存储通用的和指定的应用程序信息，支持通过Kerberos认证
3.公平调度器支持动态分级用户队列，队列是在运行时任何指定的父队列下动态创建的


2.相关包下载：
hadoop-aboutyun-linux64-2.5.2.tar.gz

链接：http://pan.baidu.com/s/1bnH94xp 密码：3fp5

hadoop-eclipse-plugin-2.5.2.jar
链接：http://pan.baidu.com/s/1pJx8llx 密码：
游客，如果您要查看本帖隐藏内容请回复


eclipse jee luna R win32 x86 64
Developers Version: Luna Release 4 4 0 Build id: 20140612 0600

hadoop2.5.2官网下载
hadoop-2.5.2-官网下载.tar.gz（下载后，名字可自行修改）
链接：http://pan.baidu.com/s/1eQpdf1K 密码：slzt


hadoop-2.5.2-src.tar(此为maven包)
链接：http://pan.baidu.com/s/1o6HneZ4 密码：m62w
hadoop-2.5.2源码获取参考
从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码


hadoop-2.5.2-api(此为文件)
链接：http://pan.baidu.com/s/1ntuo2Xr 密码：qzex

不知如何查看，可以参考如何找到hadoop2.x的api文档





编译hadoop2.5.2经验总结

编译的时候遇到了不少问题，在后面会贴出错误，参考
从零教你在Linux环境下（ubuntu 12.04）如何编译hadoop2.4环境重新安装了一遍，但是却没有预想的那么顺利，遇到了不少问题，最终归结于编译环境不纯，什么是不纯那，举个例子

当时想通过在线安装maven，可是网络实在是太慢了，所以终止在线安装，直接从网上了下载了个离线包，然后配置了一下，结果编译不成功。


最后产生了如下的问题：
问题1：

  1. [INFO] ------------------------------------------------------------------------

  2. [INFO] Reactor Summary:

  3. [INFO] 

  4. [INFO] Apache Hadoop Main ................................. SUCCESS [04:41 min]

  5. [INFO] Apache Hadoop Project POM .......................... FAILURE [32:27 min]

  6. [INFO] Apache Hadoop Annotations .......................... SKIPPED

  7. [INFO] Apache Hadoop Assemblies ........................... SKIPPED

  8. [INFO] Apache Hadoop Project Dist POM ..................... SKIPPED

  9. [INFO] Apache Hadoop Maven Plugins ........................ SKIPPED

  10. [INFO] Apache Hadoop MiniKDC .............................. SKIPPED

  11. [INFO] Apache Hadoop Auth ................................. SKIPPED

  12. [INFO] Apache Hadoop Auth Examples ........................ SKIPPED

  13. [INFO] Apache Hadoop Common ............................... SKIPPED

  14. [INFO] Apache Hadoop NFS .................................. SKIPPED

  15. [INFO] Apache Hadoop Common Project ....................... SKIPPED

  16. [INFO] Apache Hadoop HDFS ................................. SKIPPED

  17. [INFO] Apache Hadoop HttpFS ............................... SKIPPED

  18. [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED

  19. [INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED

  20. [INFO] Apache Hadoop HDFS Project ......................... SKIPPED

  21. [INFO] hadoop-yarn ........................................ SKIPPED

  22. [INFO] hadoop-yarn-api .................................... SKIPPED

  23. [INFO] hadoop-yarn-common ................................. SKIPPED

  24. [INFO] hadoop-yarn-server ................................. SKIPPED

  25. [INFO] hadoop-yarn-server-common .......................... SKIPPED

  26. [INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED

  27. [INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED

  28. [INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED

  29. [INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED

  30. [INFO] hadoop-yarn-server-tests ........................... SKIPPED

  31. [INFO] hadoop-yarn-client ................................. SKIPPED

  32. [INFO] hadoop-yarn-applications ........................... SKIPPED

  33. [INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED

  34. [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED

  35. [INFO] hadoop-yarn-site ................................... SKIPPED

  36. [INFO] hadoop-yarn-project ................................ SKIPPED

  37. [INFO] hadoop-mapreduce-client ............................ SKIPPED

  38. [INFO] hadoop-mapreduce-client-core ....................... SKIPPED

  39. [INFO] hadoop-mapreduce-client-common ..................... SKIPPED

  40. [INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED

  41. [INFO] hadoop-mapreduce-client-app ........................ SKIPPED

  42. [INFO] hadoop-mapreduce-client-hs ......................... SKIPPED

  43. [INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED

  44. [INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED

  45. [INFO] Apache Hadoop MapReduce Examples ................... SKIPPED

  46. [INFO] hadoop-mapreduce ................................... SKIPPED

  47. [INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED

  48. [INFO] Apache Hadoop Distributed Copy ..................... SKIPPED

  49. [INFO] Apache Hadoop Archives ............................. SKIPPED

  50. [INFO] Apache Hadoop Rumen ................................ SKIPPED

  51. [INFO] Apache Hadoop Gridmix .............................. SKIPPED

  52. [INFO] Apache Hadoop Data Join ............................ SKIPPED

  53. [INFO] Apache Hadoop Extras ............................... SKIPPED

  54. [INFO] Apache Hadoop Pipes ................................ SKIPPED

  55. [INFO] Apache Hadoop OpenStack support .................... SKIPPED

  56. [INFO] Apache Hadoop Client ............................... SKIPPED

  57. [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED

  58. [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED

  59. [INFO] Apache Hadoop Tools Dist ........................... SKIPPED

  60. [INFO] Apache Hadoop Tools ................................ SKIPPED

  61. [INFO] Apache Hadoop Distribution ......................... SKIPPED

  62. [INFO] ------------------------------------------------------------------------

  63. [INFO] BUILD FAILURE

  64. [INFO] ------------------------------------------------------------------------

  65. [INFO] Total time: 40:20 min

  66. [INFO] Finished at: 2014-11-23T00:36:03-08:00

  67. [INFO] Final Memory: 32M/84M

  68. [INFO] ------------------------------------------------------------------------

  69. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-javadoc-plugin:jar:2.8.1 -> org.apache.maven.shared:maven-common-artifact-filters:jar:1.3: Failed to read artifact descriptor for org.apache.maven.shared:maven-common-artifact-filters:jar:1.3: Could not transfer artifact org.apache.maven:maven-parent:pom:13 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/maven/maven-parent/13/maven-parent-13.pom from central failed: Read timed out -> [Help 1]
复制代码



问题2

  1. [INFO] Apache Hadoop Main ................................. SUCCESS [  2.484 s]

  2. [INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.025 s]

  3. [INFO] Apache Hadoop Annotations .......................... SUCCESS [  4.016 s]

  4. [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.353 s]

  5. [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.283 s]

  6. [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  5.380 s]

  7. [INFO] Apache Hadoop MiniKDC .............................. FAILURE [  0.874 s]

  8. [INFO] Apache Hadoop Auth ................................. SKIPPED

  9. [INFO] Apache Hadoop Auth Examples ........................ SKIPPED

  10. [INFO] Apache Hadoop Common ............................... SKIPPED

  11. [INFO] Apache Hadoop NFS .................................. SKIPPED

  12. [INFO] Apache Hadoop Common Project ....................... SKIPPED

  13. [INFO] Apache Hadoop HDFS ................................. SKIPPED

  14. [INFO] Apache Hadoop HttpFS ............................... SKIPPED

  15. [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED

  16. [INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED

  17. [INFO] Apache Hadoop HDFS Project ......................... SKIPPED

  18. [INFO] hadoop-yarn ........................................ SKIPPED

  19. [INFO] hadoop-yarn-api .................................... SKIPPED

  20. [INFO] hadoop-yarn-common ................................. SKIPPED

  21. [INFO] hadoop-yarn-server ................................. SKIPPED

  22. [INFO] hadoop-yarn-server-common .......................... SKIPPED

  23. [INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED

  24. [INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED

  25. [INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED

  26. [INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED

  27. [INFO] hadoop-yarn-server-tests ........................... SKIPPED

  28. [INFO] hadoop-yarn-client ................................. SKIPPED

  29. [INFO] hadoop-yarn-applications ........................... SKIPPED

  30. [INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED

  31. [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED

  32. [INFO] hadoop-yarn-site ................................... SKIPPED

  33. [INFO] hadoop-yarn-project ................................ SKIPPED

  34. [INFO] hadoop-mapreduce-client ............................ SKIPPED

  35. [INFO] hadoop-mapreduce-client-core ....................... SKIPPED

  36. [INFO] hadoop-mapreduce-client-common ..................... SKIPPED

  37. [INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED

  38. [INFO] hadoop-mapreduce-client-app ........................ SKIPPED

  39. [INFO] hadoop-mapreduce-client-hs ......................... SKIPPED

  40. [INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED

  41. [INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED

  42. [INFO] Apache Hadoop MapReduce Examples ................... SKIPPED

  43. [INFO] hadoop-mapreduce ................................... SKIPPED

  44. [INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED

  45. [INFO] Apache Hadoop Distributed Copy ..................... SKIPPED

  46. [INFO] Apache Hadoop Archives ............................. SKIPPED

  47. [INFO] Apache Hadoop Rumen ................................ SKIPPED

  48. [INFO] Apache Hadoop Gridmix .............................. SKIPPED

  49. [INFO] Apache Hadoop Data Join ............................ SKIPPED

  50. [INFO] Apache Hadoop Extras ............................... SKIPPED

  51. [INFO] Apache Hadoop Pipes ................................ SKIPPED

  52. [INFO] Apache Hadoop OpenStack support .................... SKIPPED

  53. [INFO] Apache Hadoop Client ............................... SKIPPED

  54. [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED

  55. [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED

  56. [INFO] Apache Hadoop Tools Dist ........................... SKIPPED

  57. [INFO] Apache Hadoop Tools ................................ SKIPPED

  58. [INFO] Apache Hadoop Distribution ......................... SKIPPED

  59. [INFO] ------------------------------------------------------------------------

  60. [INFO] BUILD FAILURE

  61. [INFO] ------------------------------------------------------------------------

  62. [INFO] Total time: 22.376 s

  63. [INFO] Finished at: 2014-11-23T04:20:31-08:00

  64. [INFO] Final Memory: 38M/91M

  65. [INFO] ------------------------------------------------------------------------

  66. [ERROR] Failed to execute goal on project hadoop-minikdc: Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.5.2: Failure to find org.apache.directory.server:apacheds-core:jar:2.0.0-M15 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> [Help 1]

  67. [ERROR] 

  68. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

  69. [ERROR] Re-run Maven using the -X switch to enable full debug logging.

  70. [ERROR] 

  71. [ERROR] For more information about the errors and possible solutions, please read the following articles:

  72. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

  73. [ERROR] 

  74. [ERROR] After correcting the problems, you can resume the build with the command

  75. [ERROR]   mvn <goals> -rf :hadoop-minikdc
复制代码

问题3


  1. 

  2. [INFO] ------------------------------------------------------------------------

  3. [INFO] Reactor Summary:

  4. [INFO] 

  5. [INFO] Apache Hadoop Main ................................. SUCCESS [  2.713 s]

  6. [INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.034 s]

  7. [INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.880 s]

  8. [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.411 s]

  9. [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.901 s]

  10. [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  5.805 s]

  11. [INFO] Apache Hadoop Auth ................................. FAILURE [01:09 min]

  12. [INFO] Apache Hadoop Auth Examples ........................ SKIPPED

  13. [INFO] Apache Hadoop Common ............................... SKIPPED

  14. [INFO] Apache Hadoop NFS .................................. SKIPPED

  15. [INFO] Apache Hadoop MiniKDC .............................. SKIPPED

  16. [INFO] Apache Hadoop Common Project ....................... SKIPPED

  17. [INFO] Apache Hadoop HDFS ................................. SKIPPED

  18. [INFO] Apache Hadoop HttpFS ............................... SKIPPED

  19. [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED

  20. [INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED

  21. [INFO] Apache Hadoop HDFS Project ......................... SKIPPED

  22. [INFO] hadoop-yarn ........................................ SKIPPED

  23. [INFO] hadoop-yarn-api .................................... SKIPPED

  24. [INFO] hadoop-yarn-common ................................. SKIPPED

  25. [INFO] hadoop-yarn-server ................................. SKIPPED

  26. [INFO] hadoop-yarn-server-common .......................... SKIPPED

  27. [INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED

  28. [INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED

  29. [INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED

  30. [INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED

  31. [INFO] hadoop-yarn-server-tests ........................... SKIPPED

  32. [INFO] hadoop-yarn-client ................................. SKIPPED

  33. [INFO] hadoop-yarn-applications ........................... SKIPPED

  34. [INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED

  35. [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED

  36. [INFO] hadoop-yarn-site ................................... SKIPPED

  37. [INFO] hadoop-yarn-project ................................ SKIPPED

  38. [INFO] hadoop-mapreduce-client ............................ SKIPPED

  39. [INFO] hadoop-mapreduce-client-core ....................... SKIPPED

  40. [INFO] hadoop-mapreduce-client-common ..................... SKIPPED

  41. [INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED

  42. [INFO] hadoop-mapreduce-client-app ........................ SKIPPED

  43. [INFO] hadoop-mapreduce-client-hs ......................... SKIPPED

  44. [INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED

  45. [INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED

  46. [INFO] Apache Hadoop MapReduce Examples ................... SKIPPED

  47. [INFO] hadoop-mapreduce ................................... SKIPPED

  48. [INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED

  49. [INFO] Apache Hadoop Distributed Copy ..................... SKIPPED

  50. [INFO] Apache Hadoop Archives ............................. SKIPPED

  51. [INFO] Apache Hadoop Rumen ................................ SKIPPED

  52. [INFO] Apache Hadoop Gridmix .............................. SKIPPED

  53. [INFO] Apache Hadoop Data Join ............................ SKIPPED

  54. [INFO] Apache Hadoop Extras ............................... SKIPPED

  55. [INFO] Apache Hadoop Pipes ................................ SKIPPED

  56. [INFO] Apache Hadoop OpenStack support .................... SKIPPED

  57. [INFO] Apache Hadoop Client ............................... SKIPPED

  58. [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED

  59. [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED

  60. [INFO] Apache Hadoop Tools Dist ........................... SKIPPED

  61. [INFO] Apache Hadoop Tools ................................ SKIPPED

  62. [INFO] Apache Hadoop Distribution ......................... SKIPPED

  63. [INFO] ------------------------------------------------------------------------

  64. [INFO] BUILD FAILURE

  65. [INFO] ------------------------------------------------------------------------

  66. [INFO] Total time: 01:32 min

  67. [INFO] Finished at: 2014-11-23T04:50:17-08:00

  68. [INFO] Final Memory: 42M/102M

  69. [INFO] ------------------------------------------------------------------------

  70. [ERROR] Failed to execute goal on project hadoop-auth: Could not resolve dependencies for project org.apache.hadoop:hadoop-auth:jar:2.5.2: Failure to find org.apache.directory.server:apacheds-core:jar:2.0.0-M15 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> [Help 1]

  71. [ERROR] 

  72. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

  73. [ERROR] Re-run Maven using the -X switch to enable full debug logging.

  74. [ERROR] 

  75. [ERROR] For more information about the errors and possible solutions, please read the following articles:

  76. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

  77. [ERROR] 

  78. [ERROR] After correcting the problems, you can resume the build with the command

  79. [ERROR]   mvn <goals> -rf :hadoop-auth

复制代码

上面上的含义都是找不到包，可能的原因是找不到包，
但是我的网络又是通的，所以所以肯定是maven出了问题。
最后重新部署了干净的环境。

终于编译成功















参考文档：


hadoop2.5.1汇总：最新编译 32位、64位安装、源码包、API下载及新特性等


从零教你在Linux环境下（ubuntu 12.04）如何编译hadoop2.4




相关内容：



hadoop2.6.0汇总：新增功能最新编译 32位、64位安装、源码包、API下载及部署文档

hadoop2.5.2汇总：新增功能最新编译 32位、64位安装、源码包、API、eclipse插件下载


hadoop2.5.1汇总：最新编译 32位、64位安装、源码包、API下载及新特性等

hadoop2.5.1汇总：最新编译 32位、64位安装、源码包、API下载及新特性等

新手指导：hadoop官网介绍及如何下载hadoop（2.4）各个版本与查看hadoop API介绍

从零教你在Linux环境下（ubuntu 12.04）如何编译hadoop2.4






本帖被以下淘专辑推荐:

加微信w3aboutyun，可拉入技术爱好者群
回复

使用道具 举报

自定义广告语

已有(241)人评论

跳转到指定楼层
正序浏览
wchbest 发表于 2014-11-24 09:05:59
谢谢楼主了，看看哈哈
回复

使用道具 举报

liuweidong 发表于 2014-11-24 09:18:00
楼主V5，收藏
回复

使用道具 举报

韩克拉玛寒 发表于 2014-11-24 09:27:14
楼主V587，分享了
回复

使用道具 举报

271592448 发表于 2014-11-24 11:12:53
回复

使用道具 举报

evababy 发表于 2014-11-24 11:38:11
好 ，不知道是不是真64啊
回复

使用道具 举报

iqiqi 发表于 2014-11-24 12:42:34
太好了，多谢啦！
回复

使用道具 举报

alprah 发表于 2014-11-24 14:34:34
感谢楼主分享……
回复

使用道具 举报

asdfdsafsdafdsa 发表于 2014-11-24 18:02:16
提示: 该帖被管理员或版主屏蔽
回复

使用道具 举报

12345678910... 25下一页
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
nettman

超级版主

关注

  • 2911

    主题

  • 3204

    帖子

  • 478

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000