热度 1|
启动hbase日志报错,请各位兄弟帮忙看看:我现在是4台机器,Master ,Slave1,Slave2,Slave3.其中Slave1,2,3是zk, hbase的HMaster在Master上,HRegionServer在Slave1,2,3上。hbase配置如下:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://Cluster/hbase</value><!-- Cluster是hadoop HA集群的名称 -->
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>Slave1,Slave2,Slave3</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>hbase.master</name>
<value>9000</value> <!--这里注意了,只需端口即可,不必再写主机名称了! -->
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/zookeeper</value>
<description>Property fromZooKeeper's config zoo.cfg. The directory where the snapshot isstored.</description>
</property>
</configuration>
现在启动之后 HMaster进程闪退,HRegionServer一直存在。
[root@Master logs]# jps
911 Jps
61895 DFSZKFailoverController
62070 ResourceManager
61661 NameNode
[root@Slave1 logs]# jps
35977 NameNode
35859 QuorumPeerMain
38687 HRegionServer
39592 Jps
36035 DFSZKFailoverController
[root@Slave2 logs]# jps
5620 HRegionServer
3625 JournalNode
3742 NodeManager
6073 Jps
3577 DataNode
3485 QuorumPeerMain
[root@Slave3 logs]# jps
61147 Jps
58551 QuorumPeerMain
58683 JournalNode
58800 NodeManager
60698 HRegionServer
58635 DataNode
其中HMasterde 错误日志片段:
2015-05-20 02:58:51,335 DEBUG [main-EventThread] master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitWAL/WALs%2FSlave2%2C60020%2C1432115526545-splitting%2FSlave2%252C60020%252C1432115526545.14321155
36628.meta
2015-05-20 02:58:51,493 INFO [master:Master:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=Slave2,60020,1432115526545, exception=org.apache.hadoop.hbase.NotServingRegionExceptio
n: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on Slave2,60020,1432115919978
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3680)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92)
at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:51,495 INFO [master:Master:60000] master.SplitLogManager: dead splitlog workers [Slave2,60020,1432115526545]
2015-05-20 02:58:51,500 INFO [master:Master:60000] master.SplitLogManager: hdfs://Cluster/hbase/WALs/Slave2,60020,1432115526545-splitting is empty dir, no logs to split
2015-05-20 02:58:52,832 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000001
2015-05-20 02:58:52,841 FATAL [master:Master:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-20 02:58:52,842 FATAL [master:Master:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1167)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:1015)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:684)
at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,844 INFO [master:Master:60000] master.HMaster: Aborting
015-05-20 02:58:52,857 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-2 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-1] master.DeadServer: Finished processing Slave3,60020,1432115525413
2015-05-20 02:58:52,859 DEBUG [MASTER_SERVER_OPERATIONS-Master:60000-2] master.DeadServer: Finished processing Slave2,60020,1432115526545
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-1] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-3 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,859 ERROR [MASTER_SERVER_OPERATIONS-Master:60000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.util.concurrent.RejectedExecutionException: Task ServerShutdownHandler-Master,60000,1432115916139-4 rejected from org.apache.hadoop.hbase.executor.ExecutorService$TrackingThreadPoolExecutor@2499c773[Shutti
ng down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.apache.hadoop.hbase.executor.ExecutorService$Executor.submit(ExecutorService.java:224)
at org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:148)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:326)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:209)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-05-20 02:58:52,866 DEBUG [master:Master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5bf3940a
2015-05-20 02:58:52,866 INFO [master:Master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24d704247750014
2015-05-20 02:58:52,870 INFO [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750014 closed
2015-05-20 02:58:52,870 INFO [master:Master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-05-20 02:58:52,971 INFO [Master,60000,1432115916139.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: Master,60000,1432115916139.splitLogManagerTimeoutMonitor exiting
2015-05-20 02:58:52,974 INFO [master:Master:60000] zookeeper.ZooKeeper: Session: 0x24d704247750012 closed
2015-05-20 02:58:52,974 INFO [master:Master:60000] master.HMaster: HMaster main thread exiting
2015-05-20 02:58:52,975 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3031)
2015-05-20 02:58:52,977 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
另外是HRegionServer的错误日志:
2015-05-20 03:08:41,330 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:44,333 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:47,336 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:50,339 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:53,342 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:56,344 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:08:59,347 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:02,350 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:05,353 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:08,355 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
2015-05-20 03:09:11,358 DEBUG [regionserver60020] regionserver.HRegionServer: No master found; retry
请各位兄弟,帮忙看看,愁了两天了。
我自己用下面的解决方法没有解决:
0、环境配置:hadoop2.6.0,Hbase-0.98.9,zookeeper-3.4.6.
1、将hadoop/etc/hadoop/下的core-sire.xml和hdfs-site.xml复制到hbase/conf中。
2、替换hbase/lib下的hadoop包,
3、停止hbase,删除 /tmp下的文件。
4、执行下面操作:
在zookeeper node做下面的步骤
1.运行/usr/lib/zookeeper/bin/zkCli.sh
2. ls /hbase/table-lock,如果存在,运行第三步
3.rmr /hbase/table-lock/hbase:namespace
4.重启HBase