分享

hadoop、hbase、hive版本对应关系查找表

pig2 2014-8-9 12:34:08 发表于 总结型 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 2 25905
本帖最后由 pig2 于 2014-8-9 12:47 编辑
问题导读:
网上很多同学都找不到hbase、hive、hadoop的对应关系,这里整理一下
1.hadoop2.X与hbase、哪些版本对应?
2.hadoop2.X与hive哪些版本对应?





hadoop与hive版本对应关系

hive版本0.13.1支持hadoop版本
Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y

hive版本0.13.0支持hadoop版本
Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y

hive版本0.12.0支持hadoop版本
Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y

hive版本0.11.0支持hadoop版本
Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y

hive版本0.10.0支持hadoop版本
Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y





hadoop1.2+hbase0.95.0+hive0.11.0 会产生hbase+hive的不兼容,创建hive+hbase的关联表就会报pair对异常。
hadoop1.2+hbase0.94.9+hive0.10.0 没问题,解决了上个版本的不兼容问题。

  hadoop-1.0.3+hive-0.9.0+hbase-0.92.0兼容
hadoop2.2+hbase0.96+hive0.12兼容(有些小问题,可能需要一些补丁)
hadoop2.2+hbase0.96+hive0.13兼容(应该兼容)

Hadoop 与hbase支持版本


S = 支持并且测试,
X = 不支持,
NT = 应该可以,但是没有测试.



HBase-0.92.x
HBase-0.94.x
HBase-0.96.x
HBase-0.98.x[a]
HBase-1.0.x
Hadoop-0.20.205
S
X
X
X
X
Hadoop-0.22.x
S
X
X
X
X
Hadoop-1.0.0-1.0.2[c]
X
X
X
X
X
Hadoop-1.0.3+
S
S
S
X
X
Hadoop-1.1.x
NT
S
S
X
X
Hadoop-0.23.x
X
S
NT
X
X
Hadoop-2.0.x-alpha
X
NT
X
X
X
Hadoop-2.1.0-beta
X
NT
S
X
X
Hadoop-2.2.0
X
NT [d]
S
S
NT
Hadoop-2.3.x
X
NT
S
S
NT
Hadoop-2.4.x
X
NT
S
S
S
Hadoop-2.5.x
X
NT
S
S
S


[a] Support for Hadoop 1.x is deprecated.
Hadoop 1.x is NOT supported
[c] HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.
[d] To get 0.94.x to run on hadoop 2.2.0, you need to change the hadoop 2 and protobuf versions in the pom.xml: Here is a diff with pom.xml changes:

  1. $ svn diff pom.xml
  2. Index: pom.xml
  3. ===================================================================
  4. --- pom.xml     (revision 1545157)
  5. +++ pom.xml     (working copy)
  6. @@ -1034,7 +1034,7 @@
  7.      <slf4j.version>1.4.3</slf4j.version>
  8.      <log4j.version>1.2.16</log4j.version>
  9.      <mockito-all.version>1.8.5</mockito-all.version>
  10. -    <protobuf.version>2.4.0a</protobuf.version>
  11. +    <protobuf.version>2.5.0</protobuf.version>
  12.      <stax-api.version>1.0.1</stax-api.version>
  13.      <thrift.version>0.8.0</thrift.version>
  14.      <zookeeper.version>3.4.5</zookeeper.version>
  15. @@ -2241,7 +2241,7 @@
  16.          </property>
  17.        </activation>
  18.        <properties>
  19. -        <hadoop.version>2.0.0-alpha</hadoop.version>
  20. +        <hadoop.version>2.2.0</hadoop.version>
  21.          <slf4j.version>1.6.1</slf4j.version>
  22.        </properties>
  23.        <dependencies>
复制代码

The next step is to regenerate Protobuf files and assuming that the Protobuf has been installed:
Go to the hbase root folder, using the command line;
Type the following commands:
$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/hbase.proto
$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/ErrorHandling.proto
Building against the hadoop 2 profile by running something like the following command:
$  mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests




Replace the Hadoop Bundled With HBase!Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its lib directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is critical that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.

由于管网随时更新,这里附上管网:https://hbase.apache.org/book/configuration.html#hadoop


已有(2)人评论

跳转到指定楼层
howtodown 发表于 2014-9-12 10:57:56
补充一些内容:

hive、hbase、zookeeper的关系

回复

使用道具 举报

alprah 发表于 2014-12-10 18:43:02
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条