立即注册 登录
About云-梭伦科技 返回首页

韩克拉玛寒的个人空间 https://aboutyun.com/?9721 [收藏] [复制] [分享] [RSS]

日志

HBase启动报错,请大神帮忙看看

热度 1已有 7546 次阅读2015-5-20 18:40 |个人分类:About云

启动hbase日志报错,请各位兄弟帮忙看看:我现在是4台机器,Master ,Slave1,Slave2,Slave3.其中Slave1,2,3是zk, hbase的HMaster在Master上,HRegionServer在Slave1,2,3上。hbase配置如下:

<configuration>
        <property>
          <name>hbase.rootdir</name>
          <value>hdfs://Cluster/hbase</value><!-- Cluster是hadoop HA集群的名称 -->
        </property>
        <property>
          <name>hbase.cluster.distributed</name>
          <value>true</value>
        </property>
        <property>
          <name>hbase.zookeeper.quorum</name>
          <value>Slave1,Slave2,Slave3</value>
        </property>
        <property>
          <name>dfs.replication</name>
          <value>3</value>
        </property>
        <property>
           <name>hbase.master</name>
           <value>9000</value> <!--这里注意了,只需端口即可,不必再写主机名称了! -->
        </property> 
        <property> 
           <name>hbase.zookeeper.property.dataDir</name> 
           <value>/usr/local/zookeeper</value> 
           <description>Property fromZooKeeper's config zoo.cfg. The directory where the snapshot isstored.</description> 
       </property>
</configuration>

 

现在启动之后 HMaster进程闪退,HRegionServer一直存在。

[root@Master logs]# jps
911 Jps
61895 DFSZKFailoverController
62070 ResourceManager
61661 NameNode

[root@Slave1 logs]# jps
35977 NameNode
35859 QuorumPeerMain
38687 HRegionServer
39592 Jps
36035 DFSZKFailoverController

[root@Slave2 logs]# jps
5620 HRegionServer
3625 JournalNode
3742 NodeManager
6073 Jps
3577 DataNode
3485 QuorumPeerMain

[root@Slave3 logs]# jps
61147 Jps
58551 QuorumPeerMain
58683 JournalNode
58800 NodeManager
60698 HRegionServer
58635 DataNode

 

其中HMasterde 错误日志片段:

2015-05-20 02:58:51,335 DEBUG [main-EventThread] master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitWAL/WALs%2FSlave2%2C60020%2C1432115526545-splitting%2FSlave2%252C60020%252C1432115526545.14321155
36628.meta
2015-05-20 02:58:51,493 INFO  [master:Master:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=Slave2,60020,1432115526545, exception=org.apache.hadoop.hbase.NotServingRegionExceptio
n: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on Slave2,60020,1432115919978
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3680)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92)
        at java.lang.Thread.run(Thread.java:744)

2015-05-20 02:58:51,495 INFO  [master:Master:60000] master.SplitLogManager: dead splitlog workers [Slave2,60020,1432115526545]
2015-05-20 02:58:51,500 INFO  [master:Master:60000] master.SplitLogManager: hdfs://Cluster/hbase/WALs/Slave2,60020,1432115526545-splitting is empty dir, no logs to split

2015-05-20 02:58:52,832 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 FATAL [master:Master:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-20 02:58:52,842 FATAL [master:Master:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
        at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
        at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1167)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:1015)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:684)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,844 INFO  [master:Master:60000] master.HMaster: Aborting

015-05-20 02:58:52,857 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-2 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 0]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-1] master.DeadServer: Finished processing Slave3,60020,1432115525413
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-2] master.DeadServer: Finished processing Slave2,60020,1432115526545
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-1] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-3 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-4 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,866 DEBUG [master:Master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5bf3940a
2015-05-20 02:58:52,866 INFO  [master:Master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24d704247750014
2015-05-20 02:58:52,870 INFO  [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750014 closed
2015-05-20 02:58:52,870 INFO  [master:Master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-05-20 02:58:52,971 INFO  [Master,60000,1432115916139.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: Master,60000,1432115916139.splitLogManagerTimeoutMonitor exiting
2015-05-20 02:58:52,974 INFO  [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750012 closed
2015-05-20 02:58:52,974 INFO  [master:Master:60000] master.HMaster: HMaster main thread exiting
2015-05-20 02:58:52,975 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3031)
2015-05-20 02:58:52,977 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down

另外是HRegionServer的错误日志:

2015-05-20 03:08:41,330 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:44,333 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:47,336 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:50,339 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:53,342 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:56,344 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:59,347 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:02,350 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:05,353 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:08,355 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:11,358 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry

请各位兄弟,帮忙看看,愁了两天了。

我自己用下面的解决方法没有解决:

0、环境配置:hadoop2.6.0,Hbase-0.98.9,zookeeper-3.4.6.

1、将hadoop/etc/hadoop/下的core-sire.xml和hdfs-site.xml复制到hbase/conf中。

2、替换hbase/lib下的hadoop包,

3、停止hbase,删除  /tmp下的文件。

4、执行下面操作:

在zookeeper node做下面的步骤
1.运行/usr/lib/zookeeper/bin/zkCli.sh
2. ls /hbase/table-lock,如果存在,运行第三步
3.rmr /hbase/table-lock/hbase:namespace
4.重启HBase

 

 

 


路过

雷人

握手

鲜花

鸡蛋

发表评论 评论 (1 个评论)

回复 lielies 2015-8-14 23:09
怎么处理的呢

facelist doodle 涂鸦板

您需要登录后才可以评论 登录 | 立即注册

关闭

推荐上一条 /2 下一条