HBase集群启动报错，请各位大神帮忙看看

启动hbase日志报错，请各位兄弟帮忙看看：我现在是4台机器，Master ,Slave1,Slave2,Slave3.其中Slave1，2,3是zk， hbase的HMaster在Master上，HRegionServer在Slave1,2,3上。hbase配置如下：
<configuration>
        <property>
          <name>hbase.rootdir</name>
          <value>hdfs://Cluster/hbase</value><!-- Cluster是hadoop HA集群的名称 -->
        </property>
        <property>
          <name>hbase.cluster.distributed</name>
          <value>true</value>
        </property>
        <property>
          <name>hbase.zookeeper.quorum</name>
          <value>Slave1,Slave2,Slave3</value>
        </property>
        <property>
          <name>dfs.replication</name>
          <value>3</value>
        </property>
        <property>
           <name>hbase.master</name>
           <value>9000</value> <!--这里注意了，只需端口即可，不必再写主机名称了！ -->
        </property>  
        <property>  
           <name>hbase.zookeeper.property.dataDir</name>  
           <value>/usr/local/zookeeper</value>  
           <description>Property fromZooKeeper's config zoo.cfg. The directory where the snapshot isstored.</description>  
       </property>
</configuration>

现在启动之后 HMaster进程闪退，HRegionServer一直存在。
[root@Master logs]# jps
911 Jps
61895 DFSZKFailoverController
62070 ResourceManager
61661 NameNode
[root@Slave1 logs]# jps
35977 NameNode
35859 QuorumPeerMain
38687 HRegionServer
39592 Jps
36035 DFSZKFailoverController
[root@Slave2 logs]# jps
5620 HRegionServer
3625 JournalNode
3742 NodeManager
6073 Jps
3577 DataNode
3485 QuorumPeerMain
[root@Slave3 logs]# jps
61147 Jps
58551 QuorumPeerMain
58683 JournalNode
58800 NodeManager
60698 HRegionServer
58635 DataNode

其中HMasterde 错误日志片段：
2015-05-20 02:58:51,335 DEBUG [main-EventThread] master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitWAL/WALs%2FSlave2%2C60020%2C1432115526545-splitting%2FSlave2%252C60020%252C1432115526545.14321155
36628.meta
2015-05-20 02:58:51,493 INFO  [master:Master:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=Slave2,60020,1432115526545, exception=org.apache.hadoop.hbase.NotServingRegionExceptio
n: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on Slave2,60020,1432115919978
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3680)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:51,495 INFO  [master:Master:60000] master.SplitLogManager: dead splitlog workers [Slave2,60020,1432115526545]
2015-05-20 02:58:51,500 INFO  [master:Master:60000] master.SplitLogManager: hdfs://Cluster/hbase/WALs/Slave2,60020,1432115526545-splitting is empty dir, no logs to split
2015-05-20 02:58:52,832 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 FATAL [master:Master:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-20 02:58:52,842 FATAL [master:Master:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
        at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
        at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1167)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:1015)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:684)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,844 INFO  [master:Master:60000] master.HMaster: Aborting
015-05-20 02:58:52,857 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-2 rejected from http://www.aboutyun.com/mailto:o ... tor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti]org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 0]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-1] master.DeadServer: Finished processing Slave3,60020,1432115525413
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-2] master.DeadServer: Finished processing Slave2,60020,1432115526545
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-1] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-3 rejected from [url=http://www.aboutyun.com/mailto:org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti]org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti[/url]
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-4 rejected from [url=http://www.aboutyun.com/mailto:org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti]org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti[/url]
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
        at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
        at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,866 DEBUG [master:Master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5bf3940a
2015-05-20 02:58:52,866 INFO  [master:Master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24d704247750014
2015-05-20 02:58:52,870 INFO  [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750014 closed
2015-05-20 02:58:52,870 INFO  [master:Master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-05-20 02:58:52,971 INFO  [Master,60000,1432115916139.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: Master,60000,1432115916139.splitLogManagerTimeoutMonitor exiting
2015-05-20 02:58:52,974 INFO  [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750012 closed
2015-05-20 02:58:52,974 INFO  [master:Master:60000] master.HMaster: HMaster main thread exiting
2015-05-20 02:58:52,975 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3031)
2015-05-20 02:58:52,977 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
另外是HRegionServer的错误日志：
2015-05-20 03:08:41,330 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:44,333 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:47,336 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:50,339 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:53,342 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:56,344 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:59,347 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:02,350 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:05,353 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:08,355 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:11,358 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
请各位兄弟，帮忙看看，愁了两天了。
我自己用下面的解决方法没有解决：
0、环境配置：hadoop2.6.0，Hbase-0.98.9，zookeeper-3.4.6.
1、将hadoop/etc/hadoop/下的core-sire.xml和hdfs-site.xml复制到hbase/conf中。
2、替换hbase/lib下的hadoop包，
3、停止hbase，删除  /tmp下的文件。
4、执行下面操作：
在zookeeper node做下面的步骤
1.运行/usr/lib/zookeeper/bin/zkCli.sh
2. ls /hbase/table-lock，如果存在，运行第三步
3.rmr /hbase/table-lock/hbase:namespace
4.重启HBase

JournalNode分开放到别的节点上

slave2可能出问题了
Region hbase:meta,,1 is not online on Slave2,60020,1432115919978

bob007 发表于 2015-5-26 16:56
slave2可能出问题了
Region hbase:meta,,1 is not online on Slave2,60020,1432115919978

兄弟，因为我的Hbase集群，用了 Hadoop HA集群的nameservice名称，现在不知道是什么地方出了问题。一直没有好的解决方法

韩克拉玛寒 发表于 2015-5-26 16:58
兄弟，因为我的Hbase集群，用了 Hadoop HA集群的nameservice名称，现在不知道是什么地方出了问题。一直没 ...

不知道你是否按照日志一一检查了。从这里面没有看出太多有帮助的信息。

我的配置你参考下：

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://mycluster1/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>Master2:60000</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/usr/local/hbase/tmp</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>Master2:2181,Slave1:2181</value>
</property>
</configuration>

levycui 发表于 2015-5-26 17:33
我的配置你参考下：

嗯，好我先试试

韩克拉玛寒 发表于 2015-5-27 09:26
嗯，好我先试试

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://mycluster1/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>mycluster1:60000</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/usr/local/hbase/tmp</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>Master2:2181,Slave1:2181</value>
</property>
</configuration>
这样试试

starrycheng 发表于 2015-6-3 16:27
hbase.rootdir
hdfs://mycluster1/hbase

谢谢，兄弟，已经解决。解决方案和你给我的一样