自己编写Hadoop插件：Hadoop上路_01-手动制作hadoop-eclipse-plugin插件
问题导读：1、制作hadoop-eclipse-plugin插件，需要什么环境？
2、我们如何编译呢？
static/image/hrline/4.gif
目录
一．环境准备
1.VirtualBox虚拟机：
1）安装扩展：
2）开启网络（必需连接到互联网）：
3）安装Ubuntu：
2.安装jdk，配置环境变量：
3.安装ant：
1）安装m4：
2）安装autoconf：
3）安装automake：
4）安装libtool：
5）安装ant：
4.安装eclipse：
5.解压hadoop：
二．配置hadoop
1.设置用户的操作权限： sudo chmod 777 -R hadoop
2.%hadoop%/src/contrib/eclipse-plugin/build.xml
1）取消ivy-download：
2）添加将要打包到plugin中的第三方jar包列表：
3.%hadoop%/src/contrib/build-contrib.xml ：
1）添加hadoop的version和eclipse的eclipse.home属性：
2）取消ivy-download：
4.编辑%HADOOP_HOME%/build.xml：
1）修改hadoop版本号：
2）取消ivy-download：
5.修改%hadoop%/src/contrib/eclipse-plugin/META-INF/MANIFEST.MF：
三．执行ANT
1.进入到%hadoop%/执行ant compile：
2.进入到%hadoop%/build/contrib/eclipse-plugin/执行 ant jar ：
一．环境准备：
1.VirtualBox虚拟机：
VirtualBox-4.2.8-83876-Win.exe 1）安装扩展： Oracle_VM_VirtualBox_Extension_Pack-4.2.8-83876.vbox-extpack
2）开启网络（必需连接到互联网）：
3）安装Ubuntu：
ubuntu-10.04-desktop-i386.iso
（1）安装：
（2）启用粘贴板共享和拖放支持：
（3）安装增强功能：
（4）初始化root用户：
sudo passwd root
2.安装jdk，配置环境变量：
jdk-6u24-linux-i586.bin
sudo chmod u+x jdk-6u24-linux-i586.bin
sudo -s ./jdk-6u24-linux-i586.bin
sudo gedit /etc/profile
export JAVA_HOME=%JDK%
export CLASSPATH=$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
或 jdk-7u17-linux-i586.tar.gz（ 本例使用 ）
sudo tar -xzvf jdk-7u17-linux-i586.tar.gz
sudo gedit /etc/profile
export JAVA_HOME=%JDK%
export CLASSPATH=$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
3.安装ant：
m4-1.4.16.tar.gz
autoconf-2.65.tar.gz
automake-1.13.tar.gz
libtool-2.4.tar.gz
apache-ant-1.9.0-bin.tar.gz
1）安装m4：
sudo tar -zxvf m4-1.4.16.tar.gz
cd m4-1.4.16
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
2）安装autoconf：
sudo tar -zxvf autoconf-2.65.tar.gz
cd autoconf-2.65
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
3）安装automake：sudo tar -zxvf automake-1.13.tar.gz
cd automake-1.13
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
4）安装libtool：
sudo tar -zxvf libtool-2.4.tar.gz
cd libtool-2.4
sudo ./configure --prefix=/usr/local
sudo make
sudo make install
5）安装ant：
sudo tar -zxvf apache-ant-1.9.0-bin.tar.gz
sudo gedit /etc/profile
export ANT_HOME=%ANT%
export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH
source /etc/profile
4.安装eclipse：
eclipse-SDK-4.2-linux-gtk.tar.gz
sudo tar -zxvf eclipse-SDK-4.2-linux-gtk.tar.gz
5.解压hadoop：
hadoop-1.1.2.tar.gz
（1）解压到合适的目录：
sudo tar -zxvf hadoop-1.1.2.tar.gz
（2）拷贝%hadoop%/ivy/ivy-*.jar到%ant%/lib：
sudo cp /home/hep/hadoop/ivy/ivy-2.1.0.jar /home/hep/ant/lib/
二．配置hadoop：
1.设置用户的操作权限
sudo chmod 777 -R hadoop
2.%hadoop%/src/contrib/eclipse-plugin/build.xml
1）取消ivy-download：
2）添加将要打包到plugin中的第三方jar包列表：
<!-- Override jar target to specify manifest -->
<target name="jar" depends="compile" unless="skip.contrib">
<mkdir dir="${build.dir}/lib"/>
<!-- 自定义的修改内容：begin -->
<!--
<copy file="${hadoop.root}/build/hadoop-core-${version}.jar"
tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"
todir="${build.dir}/lib" verbose="true"/>
-->
<copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-cli-1.2.jar"todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar"todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar"todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-2.4.jar"todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"todir="${build.dir}/lib" verbose="true"/>
<!-- 自定义的修改内容：end -->
<jar
jarfile="${build.dir}/hadoop-${name}-${version}.jar"
manifest="${root}/META-INF/MANIFEST.MF">
<fileset dir="${build.dir}" includes="classes/ lib/"/>
<fileset dir="${root}" includes="resources/ plugin.xml"/>
</jar>
</target><span style="font-size:10pt;line-height:1.5;font-family:'sans serif', tahoma, verdana, helvetica;"></span>
3.%hadoop%/src/contrib/build-contrib.xml ：
1）添加hadoop的version和eclipse的eclipse.home属性：
<?xml version="1.0"?>
<!-- Imported by contrib/*/build.xml files to share generic targets. -->
<project name="hadoopbuildcontrib" xmlns:ivy="antlib:org.apache.ivy.ant">
<property name="name" value="${ant.project.name}"/>
<property name="root" value="${basedir}"/>
<property name="hadoop.root" location="${root}/../../../"/>
<!-- hadoop版本、eclipse安装路径 -->
<property name="version" value="1.1.2"/>
<property name="eclipse.home" location="%eclipse%"/>
2）取消ivy-download：
4.编辑%HADOOP_HOME%/build.xml：
1）修改hadoop版本号：
2）取消ivy-download：
5.修改%hadoop%/src/contrib/eclipse-plugin/META-INF/MANIFEST.MF：
修改${HADOOP_HOME}/src/contrib/eclipse-plugin/META-INF/MANIFEST.MF的Bundle-ClassPath：
Bundle-ClassPath: classes/,
lib/hadoop-core.jar,
lib/commons-cli-1.2.jar,
lib/commons-configuration-1.6.jar,
lib/commons-httpclient-3.0.1.jar,
lib/commons-lang-2.4.jar,
lib/jackson-core-asl-1.8.8.jar,
lib/jackson-mapper-asl-1.8.8.jar
三．执行ANT：
1.进入到%hadoop%/执行ant compile：
hep@hep-ubuntu:~/hadoop$ ant compile
#此处略去N行
init:
contrib: fairscheduler
Created dir: /home/hep/hadoop/build/contrib/fairscheduler
Created dir: /home/hep/hadoop/build/contrib/fairscheduler/classes
Created dir: /home/hep/hadoop/build/contrib/fairscheduler/test
Created dir: /home/hep/hadoop/build/contrib/fairscheduler/system/classes
Created dir: /home/hep/hadoop/build/contrib/fairscheduler/examples
Created dir: /home/hep/hadoop/build/contrib/fairscheduler/test/logs
init-contrib:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
:: resolving dependencies :: org.apache.hadoop#fairscheduler;working@hep-ubuntu
confs:
found commons-logging#commons-logging;1.0.4 in maven2
found commons-collections#commons-collections;3.1 in maven2
found commons-cli#commons-cli;1.2 in maven2
found log4j#log4j;1.2.15 in maven2
found junit#junit;4.5 in maven2
found org.mortbay.jetty#jetty-util;6.1.26 in maven2
found org.mortbay.jetty#jetty;6.1.26 in maven2
found org.mortbay.jetty#servlet-api;2.5-20081211 in maven2
found org.mortbay.jetty#jsp-api-2.1;6.1.14 in maven2
found org.codehaus.jackson#jackson-core-asl;1.8.8 in maven2
found org.codehaus.jackson#jackson-mapper-asl;1.8.8 in maven2
found commons-httpclient#commons-httpclient;3.0.1 in maven2
found commons-configuration#commons-configuration;1.6 in maven2
found org.apache.commons#commons-math;2.1 in maven2
found commons-lang#commons-lang;2.4 in maven2
downloading http://repo1.maven.org/maven2/commons-collections/commons-collections/3.1/commons-collections-3.1.jar ...
..................
........................................
.........................................
....................................... (546kB)
.. (0kB)
commons-collections#commons-collections;3.1!commons-collections.jar (7793ms)
downloading http://repo1.maven.org/maven2/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar ...
................................ (131kB)
.. (0kB)
org.mortbay.jetty#jsp-api-2.1;6.1.14!jsp-api-2.1.jar (2364ms)
:: resolution report :: resolve 11591ms :: artifacts dl 10176ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| common | 15| 2 | 2 | 0 || 15| 2 |
---------------------------------------------------------------------
ivy-retrieve-common:
:: retrieving :: org.apache.hadoop#fairscheduler
confs:
15 artifacts copied, 0 already retrieved (4672kB/26ms)
DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
compile:
contrib: fairscheduler
/home/hep/hadoop/src/contrib/build-contrib.xml:191: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
Compiling 21 source files to /home/hep/hadoop/build/contrib/fairscheduler/classes
注: 某些输入文件使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
check-libhdfs-fuse:
check-libhdfs-exists:
compile:
check-contrib:
init:
contrib: gridmix
Created dir: /home/hep/hadoop/build/contrib/gridmix
Created dir: /home/hep/hadoop/build/contrib/gridmix/classes
Created dir: /home/hep/hadoop/build/contrib/gridmix/test
Created dir: /home/hep/hadoop/build/contrib/gridmix/system
Created dir: /home/hep/hadoop/build/contrib/gridmix/system/classes
Created dir: /home/hep/hadoop/build/contrib/gridmix/test/logs
init-contrib:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
:: resolving dependencies :: org.apache.hadoop#gridmix;working@hep-ubuntu
confs:
found commons-logging#commons-logging;1.0.4 in maven2
found log4j#log4j;1.2.15 in maven2
found junit#junit;4.5 in maven2
found commons-httpclient#commons-httpclient;3.0.1 in maven2
found commons-codec#commons-codec;1.4 in maven2
found commons-net#commons-net;3.1 in maven2
found org.mortbay.jetty#jetty;6.1.26 in maven2
found org.mortbay.jetty#jetty-util;6.1.26 in maven2
found org.mortbay.jetty#servlet-api;2.5-20081211 in maven2
found org.mortbay.jetty#jsp-api-2.1;6.1.14 in maven2
found org.mortbay.jetty#jsp-2.1;6.1.14 in maven2
found asm#asm;3.2 in maven2
found com.sun.jersey#jersey-core;1.8 in maven2
found com.sun.jersey#jersey-json;1.8 in maven2
found com.sun.jersey#jersey-server;1.8 in maven2
found commons-cli#commons-cli;1.2 in maven2
found org.codehaus.jackson#jackson-mapper-asl;1.8.8 in maven2
found org.codehaus.jackson#jackson-core-asl;1.8.8 in maven2
found commons-configuration#commons-configuration;1.6 in maven2
found commons-lang#commons-lang;2.4 in maven2
found commons-logging#commons-logging;1.1.1 in maven2
found commons-digester#commons-digester;1.8 in maven2
found commons-beanutils#commons-beanutils;1.7.0 in maven2
found commons-beanutils#commons-beanutils-core;1.8.0 in maven2
found org.apache.commons#commons-math;2.1 in maven2
downloading http://repo1.maven.org/maven2/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar ...
................................................................................
...........................................................................
............................................ (1000kB)
.. (0kB)
org.mortbay.jetty#jsp-2.1;6.1.14!jsp-2.1.jar (5438ms)
:: resolution report :: resolve 3147ms :: artifacts dl 5453ms
:: evicted modules:
commons-logging#commons-logging;1.0.4 by in
commons-logging#commons-logging;1.1 by in
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| common | 28| 1 | 1 | 3 || 25| 1 |
---------------------------------------------------------------------
ivy-retrieve-common:
:: retrieving :: org.apache.hadoop#gridmix
25 artifacts copied, 0 already retrieved (7872kB/77ms)
DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
接上面的代码：
compile:
contrib: gridmix
/home/hep/hadoop/src/contrib/build-contrib.xml:191: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
Compiling 44 source files to /home/hep/hadoop/build/contrib/gridmix/classes
注: /home/hep/hadoop/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/PseudoLocalFs.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
注: 某些输入文件使用了未经检查或不安全的操作。
注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
Trying to override old definition of task macro_tar
check-contrib:
init:
contrib: hdfsproxy
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/classes
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/test
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/system
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/system/classes
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/examples
Created dir: /home/hep/hadoop/build/contrib/hdfsproxy/test/logs
init-contrib:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
ivy-resolve-common:
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
:: resolving dependencies :: org.apache.hadoop#hdfsproxy;working@hep-ubuntu
confs:
found commons-httpclient#commons-httpclient;3.0.1 in maven2
found commons-logging#commons-logging;1.0.4 in maven2
found commons-cli#commons-cli;1.2 in maven2
found log4j#log4j;1.2.15 in maven2
found commons-logging#commons-logging-api;1.0.4 in maven2
found junit#junit;4.5 in maven2
found org.slf4j#slf4j-api;1.4.3 in maven2
found org.slf4j#slf4j-log4j12;1.4.3 in maven2
found xmlenc#xmlenc;0.52 in maven2
found org.mortbay.jetty#jetty;6.1.26 in maven2
found org.mortbay.jetty#jetty-util;6.1.26 in maven2
found org.mortbay.jetty#servlet-api;2.5-20081211 in maven2不知道，这是怎么回事。后面的代码，总是写上去后，点保存后总是会莫名的消失，请见谅！！
2.进入到%hadoop%/build/contrib/eclipse-plugin/执行 ant jar ：
hep@hep-ubuntu:~$ cd ~/hadoop/src/contrib/eclipse-plugin
hep@hep-ubuntu:~/hadoop/src/contrib/eclipse-plugin$ ant jar
Buildfile: /home/hep/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
init:
contrib: eclipse-plugin
init-contrib:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
:: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
:: resolving dependencies :: org.apache.hadoop#eclipse-plugin;working@hep-ubuntu
confs:
found commons-logging#commons-logging;1.0.4 in maven2
found log4j#log4j;1.2.15 in maven2
:: resolution report :: resolve 171ms :: artifacts dl 4ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| common | 2 | 0 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
ivy-retrieve-common:
:: retrieving :: org.apache.hadoop#eclipse-plugin
confs:
0 artifacts copied, 2 already retrieved (0kB/6ms)
DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
:: loading settings :: file = /home/hep/hadoop/ivy/ivysettings.xml
compile:
contrib: eclipse-plugin
/home/hep/hadoop/src/contrib/eclipse-plugin/build.xml:61: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
jar:
Created dir: /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/hadoop-core-1.1.2.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/hadoop-core.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/commons-cli-1.2.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/commons-cli-1.2.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/commons-configuration-1.6.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/commons-configuration-1.6.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/commons-httpclient-3.0.1.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/commons-httpclient-3.0.1.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/commons-lang-2.4.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/commons-lang-2.4.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/jackson-core-asl-1.8.8.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/jackson-core-asl-1.8.8.jar
Copying 1 file to /home/hep/hadoop/build/contrib/eclipse-plugin/lib
Copying /home/hep/hadoop/lib/jackson-mapper-asl-1.8.8.jar to /home/hep/hadoop/build/contrib/eclipse-plugin/lib/jackson-mapper-asl-1.8.8.jar
Building jar: /home/hep/hadoop/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.1.2.jar
BUILD SUCCESSFUL
Total time: 3 seconds
生成的插件jar就在本目录中。成品插件下载地址
页:
[1]