spring-hadoop之操作hbase

问题导读

1.配置beans.xml中如何配置hbase的？
2.hbase模板的demo，如何实现查询列族下的列的数据？
3.Spring和Hadoop是如何结合的？

　Spring对于属于java web技术的程序员都不会陌生，jdbcTemplate更是用的熟之又熟，下面我们来认识一下Spring大家庭的新成员：Spring-data-hadoop项目。Spring-hadoop这个项目应该是在Spring Data项目的一部分（Srping data其余还包括把Spring和JDBC，REST，主流的NoSQL结合起来了）。其实再一想，Spring和Hadoop结合会发生什么呢，其实就是把Hadoop组件的配置，任务部署之类的东西都统一到Spring的bean管理里去了。

具体操作如下：

1、下载并导入需要的jar包：

<classpathentry kind="lib" path="WebRoot/WEB-INF/lib/commons-configuration-1.6.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/commons-lang-2.5.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/commons-logging-1.1.1.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/dom4j-1.6.1.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/guava-11.0.2.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/hadoop-auth-2.0.0-cdh4.4.0.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/hadoop-common-2.0.0-cdh4.4.0.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/hadoop-mapreduce-client-core-2.0.0-cdh4.4.0.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/hbase-0.94.6-cdh4.4.0-security.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/log4j-1.2.17.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/protobuf-java-2.4.0a.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/slf4j-api-1.6.1.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/slf4j-log4j12-1.6.1.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-aop-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-aspects-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-beans-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-context-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-context-support-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-core-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-data-hadoop-2.0.0.RC2-cdh4.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-data-hadoop-batch-2.0.0.RC2-cdh4.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-data-hadoop-core-2.0.0.RC2-cdh4.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-data-hadoop-store-2.0.0.RC2-cdh4.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-expression-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-tx-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/spring-web-4.0.3.RELEASE.jar"/>
    <classpathentry kind="lib" path="WebRoot/WEB-INF/lib/zookeeper-3.4.5-cdh4.4.0.jar"/>
复制代码

2、配置web.xml：

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd">
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:beans.xml</param-value>
    </context-param>
    
    
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
    <!-- Spring 刷新Introspector防止内存泄露 -->
    <listener>
        <listener-class>org.springframework.web.util.IntrospectorCleanupListener</listener-class>
    </listener>
    
    <welcome-file-list>
        <welcome-file>login.jsp</welcome-file>
    </welcome-file-list>
</web-app>
复制代码

3、配置beans.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:context="http://www.springframework.org/schema/context"
        xmlns:p="http://www.springframework.org/schema/p"
        xmlns:aop="http://www.springframework.org/schema/aop"
        xmlns:hdp="http://www.springframework.org/schema/hadoop"
        xsi:schemaLocation="
            http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
            http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd
            http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-4.0.xsd
            http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop-2.0.xsd
            ">
    
    <!-- 自动扫描beans+注解功能注册 -->
    <context:component-scan base-package="net.xinhong"/>
    
    <hdp:configuration resources="classpath:/hbase-site.xml" />
    
    <hdp:hbase-configuration configuration-ref="hadoopConfiguration" />
    
    <bean id="hbaseTemplate" class="org.springframework.data.hadoop.hbase.HbaseTemplate" p:configuration-ref="hbaseConfiguration"/>
    
</beans>
复制代码

4、将hbase-site.xml配置文件拷贝到src目录下，参考内容如下：

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://nameservice1/hbase</value>
  </property>
  <property>
    <name>hbase.client.write.buffer</name>
    <value>62914560</value>
  </property>
  <property>
    <name>hbase.client.pause</name>
    <value>1000</value>
  </property>
  <property>
    <name>hbase.client.retries.number</name>
    <value>10</value>
  </property>
  <property>
    <name>hbase.client.scanner.caching</name>
    <value>1</value>
  </property>
  <property>
    <name>hbase.client.keyvalue.maxsize</name>
    <value>62914560</value>
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.security.authentication</name>
    <value>simple</value>
  </property>
  <property>
    <name>zookeeper.session.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
  <property>
    <name>zookeeper.znode.rootserver</name>
    <value>root-region-server</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>xinhong-hadoop-56,xinhong-hadoop-52,xinhong-hadoop-53</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
</configuration>
复制代码

5、hbase模板的demo:

查询列族下的列的数据：

public List<String> find(String tableName,String family,String cloumn){
        List<String> rows = hbaseTemplate.find(tableName, family,cloumn, new RowMapper<String>() {
            public String mapRow(Result result, int rowNum) throws Exception {
                return Bytes.toString(result.getRow());
            }
        });
        return rows;
    }
复制代码

查询指定行健的一列数据：

public String get(String tableName,String family,String cloumn,String rowKey){
        String context = hbaseTemplate.get(tableName, "NCEP_p_wa_2014032212_tp_006.nc", family, cloumn, new RowMapper<String>() {
            public String mapRow(Result result, int rowNum) throws Exception {
                return Bytes.toString(result.value());
              }
            });
        return context;
    }
复制代码

PS:其中RowMapper回调函数的使用方法可以参考http://www.cnblogs.com/zhangyukun/p/3685369.html来使用，用法大致相同。

转载请注明出处，期待共同进步...