分享

hbase master启动了又关闭问题总结

rsgg03 2013-12-12 10:50:58 发表于 疑问解答 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 9 45621
对于新手经常遇到这种问题,这里给大家总结一下:
错误类型1:
具体状况是开机以后 HMaster启动了一下。然后就down掉了。然后再也启动不起来。关闭hbase时候。子节点只能关闭zookeeper守护进程,不能关闭存储进程。
日志文件如下(节选部分):错误没有在网上找到过,求指导。pipe坏了该怎么弄啊。
2012-04-11 02:32:57,914 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting
2012-04-11 02:32:57,917 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: starting
2012-04-11 02:32:58,099 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: starting
2012-04-11 02:32:58,101 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: starting
2012-04-11 02:32:58,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: starting
2012-04-11 02:32:58,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: starting
2012-04-11 02:32:58,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: starting
2012-04-11 02:32:58,653 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=ubuntu,60000,1334136776012
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2012-04-11 02:32:59,037 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-04-11 02:32:59,038 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-04-11 02:32:59,038 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-04-11 02:32:59,195 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/ubuntu,60000,1334136776012 from backup master directory
2012-04-11 02:32:59,266 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/ubuntu,60000,1334136776012 already deleted, and this is not a retry
2012-04-11 02:32:59,266 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=ubuntu,60000,1334136776012
2012-04-11 02:32:59,877 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.io.IOException: Call to Ubuntu-chaiying0/192.168.17.133:9000 failed on local exception: java.io.IOException: Broken pipe
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy10.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:482)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:94)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:458)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:779)
at org.apache.hadoop.ipc.Client.call(Client.java:1047)
... 18 more
2012-04-11 02:32:59,898 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-04-11 02:32:59,898 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2012-04-11 02:32:59,898 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-04-11 02:32:59,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-04-11 02:32:59,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-04-11 02:32:59,898 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-04-11 02:32:59,899 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-04-11 02:32:59,900 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-04-11 02:32:59,904 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-04-11 02:32:59,904 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-04-11 02:33:00,034 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-04-11 02:33:00,035 INFO org.apache.zookeeper.ZooKeeper: Session: 0x136a0bf9ffc0000 closed
2012-04-11 02:33:00,035 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2012-04-11 02:33:00,035 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:152)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1637)
解决方案:
hadoop和hbase的版本有冲突,jar包需要调整一下。
hbase目录下lib中把hadoop的jar包换为和hbase向对应的版本(如0.90.5,hadoop-0.20.2等)即你安装的hadoop的jar包
如果还不行:建议如下操作
把hadoop的hadoop-core-1.0.4.jar和core-3.1.1.jar两个文件替换
错误类型2
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60000: exiting
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 60000: exiting
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000: exiting
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 22 on 60000: exiting
2013-04-28 11:37:26,533 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 25 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 26 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 28 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 29 on 60000: exiting
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60000: exiting
2013-04-28 11:37:26,535 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2013-04-28 11:37:26,534 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 on 60000: exiting
2013-04-28 11:37:26,535 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2013-04-28 11:37:26,543 INFO org.apache.zookeeper.ZooKeeper: Session: 0x23e4e9b7e8d000d closed
2013-04-28 11:37:26,543 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-04-28 11:37:26,543 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2013-04-28 11:37:26,543 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:152)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1724)
解决方案:
HMASTER重复启动
1、JPS检查HBASTER主机的进程是否重复启用
2、清理ZOOKEEPER和Hmaster的缓存


欢迎大家如about云官方群371358502,更新咨询,更新资源,随时关注

已有(9)人评论

跳转到指定楼层
fanbells 发表于 2013-12-12 11:04:37
如何清理ZOOKEEPER和Hmaster的缓存
回复

使用道具 举报

howtodown 发表于 2013-12-12 12:05:12
fanbells 发表于 2013-12-12 11:04
如何清理ZOOKEEPER和Hmaster的缓存

建议重启一下
回复

使用道具 举报

langke93 发表于 2015-6-28 19:35:51
如果都不行,看看日志
回复

使用道具 举报

ggggying12 发表于 2017-3-29 19:14:55
即你安装的hadoop的jar包在哪个目录里面啊
回复

使用道具 举报

ggggying12 发表于 2017-3-29 19:17:40
即你安装的hadoop的jar包在哪个目录里面啊
回复

使用道具 举报

qcbb001 发表于 2017-3-29 20:01:04
ggggying12 发表于 2017-3-29 19:17
即你安装的hadoop的jar包在哪个目录里面啊

自定义的,比如可以放到/usr/hbase
推荐参考
hbase 0.96整合到hadoop2.2三个节点全分布式安装高可靠文档
http://www.aboutyun.com/thread-7746-1-1.html


回复

使用道具 举报

ggggying12 发表于 2017-3-29 21:33:08
文档我看了,又有其他问题了  创建表的时候  Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
回复

使用道具 举报

langke93 发表于 2017-3-29 21:48:26
ggggying12 发表于 2017-3-29 21:33
文档我看了,又有其他问题了  创建表的时候  Check the value configured in 'zookeeper.znode.parent'. Th ...

zookeeper.znode.parent的配置是否正确。在Hbase/conf/core-site.xml中自己配置的zookeeper.znode.parent使用的是默认选项/hbase

回复

使用道具 举报

ggggying12 发表于 2017-3-30 14:57:04
langke93 发表于 2017-3-29 21:48
zookeeper.znode.parent的配置是否正确。在Hbase/conf/core-site.xml中自己配置的zookeeper.znode.parent ...

我使用的是默认的,没做修改
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条