图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Hive|数据仓库 › hbase0.96与hive0.12整合高可靠文档及问题总结

hbase0.96与hive0.12整合高可靠文档及问题总结

查看数: 124438 | 评论数: 36 | 收藏 9
关灯 | 提示：支持键盘翻页<-左 右->
帖子模式
    组图打开中，请稍候......
pig2
发布时间: 2014-5-28 15:03

正文摘要:

问题导读： 1.hive安装是否需要安装mysql？ 2.hive是否分为客户端和服务器端？ 3.hive的元数据库有哪两种？ 4.hive与hbase整合的关键是什么？ 5.hive的安装是否必须安装hadoop? 6.hive与hbase整合需要做哪些准 ...

回复

尘世随缘 发表于 2015-4-7 13:23:51
hive和hbase的整合，比如插入一条数据，是变成2份还是一份？
如果变成了2份，那么太浪费储存空间了。
求解答。。。
shiwt 发表于 2016-11-14 12:56:26
尘世随缘 发表于 2015-4-7 13:23
hive和hbase的整合，比如插入一条数据，是变成2份还是一份？
如果变成了2份，那么太浪费储存空间了。
求 ...

1份，使用hive外部表与hbase进行连接。
lilyzhang0209 发表于 2016-8-15 11:50:15
高手，咨询一下，sqoop export时，可以增量同步吗。增量删除增量加载？具体怎么设置参数呢。。
qcbb001 发表于 2016-6-1 17:18:41
limeng 发表于 2016-6-1 10:18
1. 之前这个配置是没有的，也是同样问题；
3.test存在的，hive也配置了远程连接权限，我运行报错后，进 ...

服务端需要装mysql其它不需要装mysql，只需要装hive
limeng 发表于 2016-6-1 10:18:58
easthome001 发表于 2016-5-31 18:18
1.
        hive.metastore.local
        false

1. 之前这个配置是没有的，也是同样问题；
3.test存在的，hive也配置了远程连接权限，我运行报错后，进入test数据库，发现已经创建了好几张表，之前是空的，

我想问下 我这样启动对吗？  我只在master上装hive，其余两个节点没装，就是在slave2上装了MySQL，其余节点需要装hive吗？
你是咋装的?    谢谢啦
limeng 发表于 2016-6-1 09:33:57
einhep 发表于 2016-5-31 18:10
已经启动一次了，后面打开客户端就可以了。
如果再次启动端口会因为被暂用，而连接不上

我不太明白你的意思，我在master上先运行命令 hive  --service metastore -hiveconf hive.root.logger=DEBUG,console 这行命令是开启metastore服务，
卡住不动之后，重新在master上开一个终端输入命令  hive  ，这个不就是开客户端吗?   
我只在master上安装了hive，其余两个节点没有安装， 只在slave2上装了MySQL.


easthome001 发表于 2016-5-31 18:18:14
einhep 发表于 2016-5-31 18:10
已经启动一次了，后面打开客户端就可以了。
如果再次启动端口会因为被暂用，而连接不上

1.<property>
        <name>hive.metastore.local</name>
        <value>false</value>
</property>
可以去掉

2.里面配置的目录查看权限

3.test是否存在
4.hive用户是否有权限远程连接

einhep 发表于 2016-5-31 18:10:04
limeng 发表于 2016-5-31 16:54
哪位大神帮忙看看问题
环境: hadoop2.2.0   192.168.0.19 master   ;  ~.~.~.124 slave1 ; ~~~.125 slave2 ...

已经启动一次了，后面打开客户端就可以了。
如果再次启动端口会因为被暂用，而连接不上
limeng 发表于 2016-5-31 16:54:52
哪位大神帮忙看看问题
环境: hadoop2.2.0   192.168.0.19 master   ;  ~.~.~.124 slave1 ; ~~~.125 slave2
hbase0.96  部署成功
在master上按照文档安装了hive0.12  采用远程模式   MySQL安装在slave2上   在master上测试可以远程连接MySQL
之后再master终端命令上  hive  --service metastore -hiveconf hive.root.logger=DEBUG,console   显示大量info后 卡住不动  
之后再master上 重新打开一个终端  输入命令  hive  然后报错了 错误如下:
[hadoop@Master ~]$ hive
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/05/31 01:16:11 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Logging initialized using configuration in jar:file:/usr/hive/lib/hive-common-0.13.0-SNAPSHOT.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:295)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:679)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
 at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1345)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2420)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2432)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:289)
... 7 more
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1343)
... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
 at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:288)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1343)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2420)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2432)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:289)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:679)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 19 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:334)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
... 17 more

master上的hive-site.xml文件如下:
<configuration>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>hdfs://Master:8020/hive/warehouse</value>
</property>
<property>
  <name>hive.exec.scratchdir</name>
  <value>hdfs://Master:8020/hive/scratchdir</value>
</property>
<property>
        <name>hive.metastore.local</name>
        <value>false</value>
</property>
<property>
        <name>hive.querylog.location</name>
        <value>/usr/hive/logs</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.0.125:3306/test?createDatabaseIfNotExist=true</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive</value>
</property>
<property>
  <name>hive.aux.jars.path</name>
  <value>file:///usr/hive/lib/hive-hbase-handler-0.13.0-SNAPSHOT.jar,file:///usr/hive/lib/protobuf-java-2.5.0.jar,file:///usr/hive/lib/hbase-client-0.96.0-hadoop2.jar,file:///usr/hive/lib/hbase-common-0.96.0-hadoop2.jar,file:///usr/hive/lib/zookeeper-3.4.5.jar,file:///usr/hive/lib/guava-11.0.2.jar</value>
</property>
<property>
        <name>hive.metastore.uris</name>
        <value>thrift://192.168.0.125:9083</value>
</property>
</configuration>

哪位大神帮帮我  谢谢!

limeng 发表于 2016-5-31 16:39:06
遇到问题 问问大家
环境如下:  hadoop(2.2.0)  192.168.0.19  master ; 192.168.0.124 slave1 ; 192.168.0.125 salve2
hbase(0.96)部署成功
我在master上参照文档安装了hive(0.12)   采用远程模式 mysql安装在salve2上,并设置了远程连接,在master上可以远程连接上MySQL
hive-site.xml文件如下:
<configuration>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>hdfs://Master:8020/hive/warehouse</value>
</property>
<property>
  <name>hive.exec.scratchdir</name>
  <value>hdfs://Master:8020/hive/scratchdir</value>
</property>
<property>
        <name>hive.metastore.local</name>
        <value>false</value>
</property>
<property>
        <name>hive.querylog.location</name>
        <value>/usr/hive/logs</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.0.125:3306/test?createDatabaseIfNotExist=true</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive</value>
</property>

<property>
  <name>hive.aux.jars.path</name>
  <value>file:///usr/hive/lib/hive-hbase-handler-0.13.0-SNAPSHOT.jar,file:///usr/hive/lib/protobuf-java-2.5.0.jar,file:///usr/hive/lib/hbase-client-0.96.0-hadoop2.jar,file:///usr/hive/lib/hbase-common-0.96.0-hadoop2.jar,file:///usr/hive/lib/zookeeper-3.4.5.jar,file:///usr/hive/lib/guava-11.0.2.jar</value>
</property>
<property>
        <name>hive.metastore.uris</name>
        <value>thrift://192.168.0.125:9083</value>
</property>
</configuration>

之后在master终端命令:hive  --service metastore -hiveconf hive.root.logger=DEBUG,console  显示很多信息后卡住不动  
之后在master上另外打开一个终端 输入命令hive
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/05/31 01:16:11 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/05/31 01:16:11 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.

Logging initialized using configuration in jar:file:/usr/hive/lib/hive-common-0.13.0-SNAPSHOT.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:295)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:679)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1345)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2420)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2432)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:289)
        ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1343)
        ... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:288)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1343)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2420)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2432)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:289)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:679)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 19 more
)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:334)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
        ... 17 more
我找了好多资料 还是没有解决，请帮帮我 谢谢
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000