问题导读:
1.如何通过官网src包,获取hadoop的全部代码
2.通过什么样的操作,可以查看hadoop某个函数或则类的实现?
3.maven的作用是什么?
我们如果想搞开发,研究源码对我们的帮助很大。不明白原理就如同黑盒子,遇到问题,我们也摸不着思路。所以这里交给大家
一.如何获取源码
二.如何关联源码
一.如何获取源码
1.下载hadoop的maven程序包
(1)官网下载
这里我们先从官网上下载maven包hadoop-2.4.0-src.tar.gz。
官网下载地址
对于不知道怎么去官网下载,可以查看:新手指导:hadoop官网介绍及如何下载hadoop(2.4)各个版本与查看hadoop API介绍
(2)网盘下载
也可以从网盘下载:
http://pan.baidu.com/s/1kToPuGB
2.通过maven获取源码
获取源码的方式有两种,一种是通过命令行的方式,一种是通过eclipse。这里主要讲通过命令的方式
通过命令的方式获取源码:
1.解压包
解压包的时候遇到了下面问题。不过不用管,我们继续往下走
1 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\ApplicationHistoryClientService$ApplicationHSClientProtocolHandler.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
2 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\timeline\LeveldbTimelineStore$LockMap$CountingReentrantLock.class:系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
3 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\test-classes\org\apache\hadoop\yarn\server\applicationhistoryservice\webapp\TestAHSWebApp$MockApplicationHistoryManagerImpl.class:系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
4 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\monitor\capacity\TestProportionalCapacityPreemptionPolicy$IsPreemptionRequestFor.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
5 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestFSRMStateStore$TestFSRMStateStoreTester$TestFileSystemRMStore.class:系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
6 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStore$TestZKRMStateStoreTester$TestZKRMStateStoreInternal.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
7 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestForwardingWatcher.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
8 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestZKRMStateStore.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
9 : 无法创建文件:D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\rmapp\attempt\TestRMAppAttemptTransitions$TestApplicationAttemptEventDispatcher.class:
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。 D:\hadoop2\hadoop-2.4.0-src.zip
2.通过maven获取源码
这里需要说明的是,在使用maven的时候,需要先安装jdk,protoc ,如果没有安装可以参考win7如何安装maven、安装protoc
(1)进入hadoop-2.4.0-src\hadoop-maven-plugins,运行mvn install
D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins>mvn install 复制代码
显示如下信息
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found d
uplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @
org.apache.hadoop:hadoop-project:2.4.0, D:\hadoop2\hadoop-2.4.0-src\hadoop-proje
ct\pom.xml, line 1015, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten t
he stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support buildin
g such malformed projects.
[WARNING]
[INFO]
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethrea
ded.SingleThreadedBuilder with a thread count of 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Maven Plugins 2.4.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins
---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave
n-plugins ---
[INFO] Using 'UTF-8' encoding to read mojo metadata.
[INFO] Applying mojo extractor for language: java-annotations
[INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
[INFO] Applying mojo extractor for language: java
[INFO] Mojo extractor for language: java found 0 mojo descriptors.
[INFO] Applying mojo extractor for language: bsh
[INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
[INFO]
[INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-mav
en-plugins ---
[INFO] Using default encoding to copy filtered resources.
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-
plugins ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p
lugins ---
[INFO] Using 'UTF-8' encoding to read mojo metadata.
[INFO] Applying mojo extractor for language: java-annotations
[INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
[INFO] Applying mojo extractor for language: java
[INFO] Mojo extractor for language: java found 0 mojo descriptors.
[INFO] Applying mojo extractor for language: bsh
[INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
[INFO]
[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ ha
doop-maven-plugins ---
[INFO] Using default encoding to copy filtered resources.
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoo
p-maven-plugins ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins
---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins ---
[INFO] Building jar: D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\had
oop-maven-plugins-2.4.0.jar
[INFO]
[INFO] --- maven-plugin-plugin:3.0:addPluginArtifactMetadata (default-addPluginA
rtifactMetadata) @ hadoop-maven-plugins ---
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-
maven-plugins ---
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p
lugins ---
[INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\hadoop
-maven-plugins-2.4.0.jar to C:\Users\hyj\.m2\repository\org\apache\hadoop\hadoop
-maven-plugins\2.4.0\hadoop-maven-plugins-2.4.0.jar
[INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\pom.xml to C:
\Users\hyj\.m2\repository\org\apache\hadoop\hadoop-maven-plugins\2.4.0\hadoop-ma
ven-plugins-2.4.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.891 s
[INFO] Finished at: 2014-06-23T14:47:33+08:00
[INFO] Final Memory: 21M/347M
[INFO] ------------------------------------------------------------------------ 复制代码
部分截图如下:
(2)运行
mvn eclipse:eclipse -DskipTests 复制代码
这时候注意,我们进入的是hadoop_home,我这里是D:\hadoop2\hadoop-2.4.0-src
部分信息如下
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [ 0.684 s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [ 0.720 s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [ 0.276 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [ 0.179 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [ 0.121 s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [ 1.680 s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [ 1.802 s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [ 1.024 s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [ 0.160 s]
[INFO] Apache Hadoop Common .............................. SUCCESS [ 1.061 s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [ 0.489 s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [ 0.056 s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [ 2.770 s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [ 0.965 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [ 0.629 s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [ 0.284 s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [ 0.061 s]
[INFO] hadoop-yarn ....................................... SUCCESS [ 0.052 s]
[INFO] hadoop-yarn-api ................................... SUCCESS [ 0.842 s]
[INFO] hadoop-yarn-common ................................ SUCCESS [ 0.322 s]
[INFO] hadoop-yarn-server ................................ SUCCESS [ 0.065 s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [ 0.972 s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [ 0.580 s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [ 0.379 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [ 0.281 s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [ 0.378 s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [ 0.534 s]
[INFO] hadoop-yarn-client ................................ SUCCESS [ 0.307 s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [ 0.050 s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [ 0.202 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [ 0.194 s]
[INFO] hadoop-yarn-site .................................. SUCCESS [ 0.057 s]
[INFO] hadoop-yarn-project ............................... SUCCESS [ 0.066 s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [ 0.091 s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [ 1.321 s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [ 0.786 s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [ 0.456 s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [ 0.508 s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [ 0.834 s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [ 0.541 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [ 0.284 s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [ 0.851 s]
[INFO] hadoop-mapreduce .................................. SUCCESS [ 0.099 s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [ 0.742 s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [ 0.335 s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [ 0.397 s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [ 0.371 s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [ 0.230 s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [ 0.184 s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [ 0.217 s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [ 0.048 s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [ 0.244 s]
[INFO] Apache Hadoop Client .............................. SUCCESS [ 0.590 s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [ 0.230 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [ 0.650 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [ 0.334 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [ 0.042 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 0.144 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31.234 s
[INFO] Finished at: 2014-06-23T14:55:08+08:00
[INFO] Final Memory: 84M/759M
[INFO] ------------------------------------------------------------------------ 复制代码
这时候,我们已经把源码给下载下来了。这时候,我们会看到文件会明显增大。
3.关联eclipse源码
加入我们以下程序
hadoop2.2mapreduce例子.rar
(1.14 MB, 下载次数: 549, 售价: 1 云币)