hadoop2.7新增功能
版本更新说明:
http://hadoop.apache.org/docs/r2.7.0/
Common
当使用HTTP proxy server时,认证有所加强.这是对于使用proxy server访问WebHDFS是非常有用的
一种hadoop新的允许直接写Graphite的标准库
规范了HCFS(Hadoop Compatible Filesystem )的相关帮助
HDFS
支持POSIX风格的FileSystem扩展属性。.更多查看用户文档
使用 OfflineImageViewer, 客户端可以通过 WebHDFS API查看fsimage.
NFS网关做了些改进和bug修复。 Hadoop portmapper 不在需要运行网关, 网关可以拒绝非特权端口连接。
SecondaryNameNode, JournalNode, and DataNode 已经可以可视化
YARN
YARN’s REST APIs 现在支持 写/修改 操作. 用户可以通过REST APIs提交和 kill应用程序
时间存储在yarn中,为applications的存储和应用的通用具体信息,支持通过Kerberos认证。
公平调度器支持动态的按等级划分队列,用户队列可以动态的创建在其它父队列里
安装部署:
hadoop2.7集群完全分布式安装
hadoop2.7【单节点】单机、伪分布、分布式安装指导
下载地址
http://apache.fayea.com/hadoop/common/hadoop-2.7.0/
官网下载安装包
hadoop-2.7.0.tar.gz
链接:http://pan.baidu.com/s/1eQ1sUjS 密码:0h97
maven包
hadoop-2.7.0-src.tar
链接:http://pan.baidu.com/s/1pJ7gmlt 密码:xqyw
64位编译包
hadoop-2.7.0-aboutyun-64bit.tar.gz
链接:http://pan.baidu.com/s/1o6kNDGY 密码:jpuj
hadoop2.7-api文档
链接:http://pan.baidu.com/s/1kTDZYPD 密码:j1r9
eclipse插件 Linux及window操作系统都不太一样,导致插件使用不了,
最好自己制作适合自己的插件
如果真不会制作也有办法开发,可查看
hadoop开发方式总结及操作指导
############################################
编译64位包[官网已支持64位]
1.编译按照这篇帖子即可成功:
从零教你在Linux环境下(ubuntu)如何编译hadoop2.4
最好在root下编译,否则可能会遇到权限问题
2.总结编译中遇到的两大问题:
1.插件下载不完全
表现:Could not resolve dependencies
例如:
Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0:
解决办法:
这种情况很常见,而且很多都碰到了,他们也是完全按照文档来配置的,但是就不成功,这就是因为插件没有下载完毕造成的。所以尽量多执行几次下面命令
mvn package -Pdist,native -DskipTests -Dtar 复制代码
2.权限问题
权限一般报找不到文件
表现
No such file or directory
例如:
Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml (No such file or directory ) -> [Help 1]
解决办法:
在root下编译。然后确保所属权限属于root。
剩下的就是多编译几次
3.相关工具:
1.vmware10
链接:http://pan.baidu.com/s/1bIBuA 密码:3ss9
2.winscp516setup
链接:http://pan.baidu.com/s/1nt64rzj 密码:2109
3.jdk-7u67-linux-x64.tar
链接:http://pan.baidu.com/s/1pJ60syf 密码:9kee
4.protobuf-2.5.0.tar.gz
链接:http://pan.baidu.com/s/1dDgWe6P 密码:xgiu
5.apache-ant-1.9.4-bin.tar.gz
http://pan.baidu.com/s/1c0vjhBy
4.编译遇到问题
1.权限问题
本来已经改变了文件夹的权限,可是文件夹里面的还是没有改变
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [4:34.701s]
[INFO] Apache Hadoop Project POM ......................... FAILURE [29.710s]
[INFO] Apache Hadoop Annotations ......................... SKIPPED
[INFO] Apache Hadoop Assemblies .......................... SKIPPED
[INFO] Apache Hadoop Project Dist POM .................... SKIPPED
[INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
[INFO] Apache Hadoop MiniKDC ............................. SKIPPED
[INFO] Apache Hadoop Auth ................................ SKIPPED
[INFO] Apache Hadoop Auth Examples ....................... SKIPPED
[INFO] Apache Hadoop Common .............................. SKIPPED
[INFO] Apache Hadoop NFS ................................. SKIPPED
[INFO] Apache Hadoop KMS ................................. SKIPPED
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-registry .............................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
[INFO] Apache Hadoop Azure support ....................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:34.724s
[INFO] Finished at: Wed Apr 22 03:40:41 PDT 2015
[INFO] Final Memory: 28M/67M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml (No such file or directory) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-project
查看下权限,都是root
aboutyun@ubuntu:~/hadoop-2.7.0-src$ ll 复制代码
total 120
drwxr-xr-x 15 aboutyun aboutyun 4096 Apr 10 11:39 ./
drwxr-xr-x 24 aboutyun aboutyun 4096 Apr 22 03:32 ../
-rw-r--r-- 1 root root 13110 Apr 10 11:35 BUILDING.txt
drwxr-xr-x 2 root root 4096 Apr 22 03:31 dev-support/
drwxr-xr-x 3 root root 4096 Apr 22 03:31 hadoop-assemblies/
drwxr-xr-x 2 root root 4096 Apr 22 03:31 hadoop-client/
drwxr-xr-x 10 root root 4096 Apr 22 03:31 hadoop-common-project/
drwxr-xr-x 2 root root 4096 Apr 22 03:31 hadoop-dist/
drwxr-xr-x 6 root root 4096 Apr 22 03:31 hadoop-hdfs-project/
drwxr-xr-x 9 root root 4096 Apr 22 03:31 hadoop-mapreduce-project/
drwxr-xr-x 3 root root 4096 Apr 22 03:31 hadoop-maven-plugins/
drwxr-xr-x 2 root root 4096 Apr 22 03:31 hadoop-minicluster/
drwxr-xr-x 3 root root 4096 Apr 22 03:31 hadoop-project/
drwxr-xr-x 2 root root 4096 Apr 22 03:31 hadoop-project-dist/
drwxr-xr-x 16 root root 4096 Apr 22 03:31 hadoop-tools/
drwxr-xr-x 3 root root 4096 Apr 22 03:31 hadoop-yarn-project/
-rw-r--r-- 1 root root 15429 Apr 10 11:35 LICENSE.txt
-rw-r--r-- 1 root root 101 Apr 10 11:35 NOTICE.txt
-rw-r--r-- 1 root root 18893 Apr 10 11:35 pom.xml
-rw-r--r-- 1 root root 1366 Apr 10 11:35 README.txt 复制代码
修改为aboutyun.
执行下面命令:
sudo chown aboutyun:aboutyun hadoop-2.7.0-src/*
复制代码
修改完毕
错误2
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.4.4/plexus-archiver-2.4.4.jar (161 KB at 34.7 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [2.569s]
[INFO] Apache Hadoop Project POM ......................... FAILURE [2:16.305s]
[INFO] Apache Hadoop Annotations ......................... SKIPPED
[INFO] Apache Hadoop Assemblies .......................... SKIPPED
[INFO] Apache Hadoop Project Dist POM .................... SKIPPED
[INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
[INFO] Apache Hadoop MiniKDC ............................. SKIPPED
[INFO] Apache Hadoop Auth ................................ SKIPPED
[INFO] Apache Hadoop Auth Examples ....................... SKIPPED
[INFO] Apache Hadoop Common .............................. SKIPPED
[INFO] Apache Hadoop NFS ................................. SKIPPED
[INFO] Apache Hadoop KMS ................................. SKIPPED
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-registry .............................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
[INFO] Apache Hadoop Azure support ....................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:25.054s
[INFO] Finished at: Wed Apr 22 03:50:50 PDT 2015
[INFO] Final Memory: 30M/71M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) on project hadoop-project: Execution hadoop-java-sources of goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork failed: Plugin org.apache.maven.plugins:maven-source-plugin:2.3 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.reporting:maven-reporting-api:jar:2.0.6 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar from central failed: Read timed out -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-project
复制代码
执行了下复制代码
第二个过了,后面又遇到了问题,文件夹里,还有文件里,而且还是root,最后直接切换root,免得在遇到权限问题
切换root遇到问题
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.hadoop:hadoop-main:2.7.0 (/root/hadoop-2.7.0-src/pom.xml) has 1 error
[ERROR] Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.5.0 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.felix:maven-bundle-plugin:jar:2.5.0 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.6: Could not transfer artifact org.apache.maven:maven-plugin-parameter-documenter:pom:2.0.6 from/to central (http://repo.maven.apache.org/maven2): Read timed out -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
复制代码
又遇到[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hadoop-minikdc: Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0: The following artifacts could not be resolved: org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15, org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20, org.apache.directory.api:api-ldap-model:jar:1.0.0-M20, org.apache.directory.api:api-util:jar:1.0.0-M20, org.apache.mina:mina-core:jar:2.0.0-M5, net.sf.ehcache:ehcache-core:jar:2.4.4, org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15: Could not transfer artifact org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/directory/server/apacheds-core-api/2.0.0-M15/apacheds-core-api-2.0.0-M15.jar from central failed: Read timed out -> [Help 1]
复制代码
总结:
遇到这种情况,一般都是插件没有下载
Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0: 复制代码
可以多次执行,下载完毕。
参考:
从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4