日志
master 闪退
热度 1 |
2015-08-15 11:42:07,438 INFO [master:bigdate1:60000-WAL.AsyncSyncer4] wal.FSHLog: master:bigdate1:60000-WAL.AsyncSyncer4 exiting
2015-08-15 11:42:07,438 DEBUG [master:bigdate1:60000-WAL.AsyncWriter] wal.FSHLog: master:bigdate1:60000-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer
2015-08-15 11:42:07,438 INFO [master:bigdate1:60000-WAL.AsyncWriter] wal.FSHLog: master:bigdate1:60000-WAL.AsyncWriter exiting
2015-08-15 11:42:07,438 DEBUG [master:bigdate1:60000] wal.FSHLog: Closing WAL writer in hdfs://bigdate1:8020/hbase/data/hbase/meta/1588230740/WALs
2015-08-15 11:42:07,558 DEBUG [master:bigdate1:60000] wal.FSHLog: Moved 1 WAL file(s) to /hbase/data/hbase/meta/1588230740/oldWALs
2015-08-15 11:42:07,769 DEBUG [master:bigdate1:60000] util.FSTableDescriptors: Wrote descriptor into: hdfs://bigdate1:8020/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2015-08-15 11:42:07,898 INFO [master:bigdate1:60000] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2015-08-15 11:42:07,927 DEBUG [master:bigdate1:60000] master.SplitLogManager: Distributed log replay=false, hfile.format.version=2
2015-08-15 11:42:07,936 INFO [master:bigdate1:60000] master.SplitLogManager: Timeout=120000, unassigned timeout=180000, distributedLogReplay=false
2015-08-15 11:42:07,941 INFO [master:bigdate1:60000] master.SplitLogManager: Found 0 orphan tasks and 0 rescan nodes
2015-08-15 11:42:07,942 DEBUG [master:bigdate1:60000] util.FSTableDescriptors: Fetching table descriptors from the filesystem.
2015-08-15 11:42:08,017 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x700a7ec connecting to ZooKeeper ensemble=bigdate5:2181,bigdate4:2181,bigdate6:2181
2015-08-15 11:42:08,017 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=bigdate5:2181,bigdate4:2181,bigdate6:2181 sessionTimeout=90000 watcher=hconnection-0x700a7ec, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:08,019 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Opening socket connection to server bigdate5/172.19.21.105:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-15 11:42:08,020 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Socket connection established to bigdate5/172.19.21.105:2181, initiating session
2015-08-15 11:42:08,034 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Session establishment complete on server bigdate5/172.19.21.105:2181, sessionid = 0x24f2f58d028000c, negotiated timeout = 40000
2015-08-15 11:42:08,092 DEBUG [master:bigdate1:60000] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@39f52df3
2015-08-15 11:42:08,214 INFO [master:bigdate1:60000] master.HMaster: Server active/primary master=bigdate1,60000,1439610118161, sessionid=0x24f2f58d0280005, setting cluster-up flag (Was=true)
2015-08-15 11:42:08,249 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Node /hbase/online-snapshot/acquired already exists and this is not a retry
2015-08-15 11:42:08,252 INFO [master:bigdate1:60000] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort
2015-08-15 11:42:08,256 DEBUG [master:bigdate1:60000] procedure.ZKProcedureCoordinatorRpcs: Starting the controller for procedure member:bigdate1,60000,1439610118161
2015-08-15 11:42:08,310 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_OPEN_REGION-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,310 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_CLOSE_REGION-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_SERVER_OPERATIONS-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_META_SERVER_OPERATIONS-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=M_LOG_REPLAY_OPS-bigdate1:60000, corePoolSize=10, maxPoolSize=10
2015-08-15 11:42:08,312 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_TABLE_OPERATIONS-bigdate1:60000, corePoolSize=1, maxPoolSize=1
2015-08-15 11:42:08,315 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner
2015-08-15 11:42:08,321 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=bigdate5:2181,bigdate4:2181,bigdate6:2181
2015-08-15 11:42:08,321 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=bigdate5:2181,bigdate4:2181,bigdate6:2181 sessionTimeout=90000 watcher=replicationLogCleaner, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:08,323 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Opening socket connection to server bigdate5/172.19.21.105:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-15 11:42:08,327 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Socket connection established to bigdate5/172.19.21.105:2181, initiating session
2015-08-15 11:42:08,333 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Session establishment complete on server bigdate5/172.19.21.105:2181, sessionid = 0x24f2f58d028000d, negotiated timeout = 40000
2015-08-15 11:42:08,343 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Node /hbase/replication/rs already exists and this is not a retry
2015-08-15 11:42:08,343 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
2015-08-15 11:42:08,349 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner
2015-08-15 11:42:08,353 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner
2015-08-15 11:42:08,355 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner
2015-08-15 11:42:08,357 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner
2015-08-15 11:42:08,357 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:08,792 INFO [FifoRpcScheduler.handler1-thread-4] master.ServerManager: Registering server=bigdate4,60020,1439610125637
2015-08-15 11:42:08,796 INFO [FifoRpcScheduler.handler1-thread-4] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-08-15 11:42:08,825 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 468 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:08,843 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,654 INFO [FifoRpcScheduler.handler1-thread-5] master.ServerManager: Registering server=bigdate2,60020,1439610123562
2015-08-15 11:42:09,657 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 2, slept for 1300 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:09,679 INFO [FifoRpcScheduler.handler1-thread-6] master.ServerManager: Registering server=bigdate3,60020,1439610125094
2015-08-15 11:42:09,687 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,688 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:09,709 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 1352 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:09,739 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,740 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:09,741 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:10,601 INFO [FifoRpcScheduler.handler1-thread-8] master.ServerManager: Registering server=bigdate1,60020,1439610119732
2015-08-15 11:42:10,634 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:10,636 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:10,638 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:10,640 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:10,645 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 4, slept for 2288 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:11,235 INFO [FifoRpcScheduler.handler1-thread-10] master.ServerManager: Registering server=bigdate5,60020,1439610126975
2015-08-15 11:42:11,269 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 5, slept for 2912 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:11,296 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:11,298 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:11,300 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:11,302 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:11,304 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate5,60020,1439610126975
2015-08-15 11:42:12,778 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 5, slept for 4421 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:12,882 INFO [master:bigdate1:60000] master.ServerManager: Finished waiting for region servers count to settle; checked in 5, slept for 4525 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-08-15 11:42:12,891 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate1,60020,1439610119732 belongs to an existing region server
2015-08-15 11:42:12,891 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate2,60020,1439610123562 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate3,60020,1439610125094 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate4,60020,1439610125637 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate5,60020,1439610126975 belongs to an existing region server
2015-08-15 11:42:13,105 INFO [master:bigdate1:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=bigdate2,60020,1439559068496, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on bigdate2,60020,1439610123562
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3680)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92)
at java.lang.Thread.run(Thread.java:745)
2015-08-15 11:42:13,110 INFO [master:bigdate1:60000] master.MasterFileSystem: Log dir for server bigdate2,60020,1439559068496 does not exist
2015-08-15 11:42:13,110 INFO [master:bigdate1:60000] master.SplitLogManager: dead splitlog workers [bigdate2,60020,1439559068496]
2015-08-15 11:42:13,111 DEBUG [master:bigdate1:60000] master.SplitLogManager: Scheduling batch of logs to split
2015-08-15 11:42:13,112 INFO [master:bigdate1:60000] master.SplitLogManager: started splitting 0 logs in []
2015-08-15 11:42:13,113 INFO [master:bigdate1:60000] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [] in 1ms
2015-08-15 11:42:13,114 INFO [master:bigdate1:60000] zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in ZooKeeper
2015-08-15 11:42:13,129 DEBUG [master:bigdate1:60000] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=bigdate5,60020,1439610126975; 5 (online=5, available=5) available servers, forceNewPlan=false
2015-08-15 11:42:13,129 DEBUG [master:bigdate1:60000] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state
2015-08-15 11:42:13,153 INFO [master:bigdate1:60000] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to bigdate5,60020,1439610126975
2015-08-15 11:42:13,153 INFO [master:bigdate1:60000] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1439610133129, server=null} to {1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,153 DEBUG [master:bigdate1:60000] master.ServerManager: New admin connection to bigdate5,60020,1439610126975
2015-08-15 11:42:13,350 INFO [master:bigdate1:60000] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-08-15 11:42:13,413 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager: Handling RS_ZK_REGION_OPENING, server=bigdate5,60020,1439610126975, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,416 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975} to {1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,537 INFO [FifoRpcScheduler.handler1-thread-15] master.ServerManager: Registering server=bigdate6,60020,1439610131812
2015-08-15 11:42:13,578 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:13,584 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:13,586 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:13,588 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:13,590 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate6,60020,1439610131812
2015-08-15 11:42:13,592 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate5,60020,1439610126975
2015-08-15 11:42:13,994 DEBUG [AM.ZK.Worker-pool2-t2] master.AssignmentManager: Handling RS_ZK_REGION_OPENED, server=bigdate5,60020,1439610126975, region=1588230740, current_state={1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,995 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975} to {1588230740 state=OPEN, ts=1439610133995, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:14,000 INFO [AM.ZK.Worker-pool2-t2] handler.OpenedRegionHandler: Handling OPENED of 1588230740 from bigdate5,60020,1439610126975; deleting unassigned node
2015-08-15 11:42:14,011 DEBUG [AM.ZK.Worker-pool2-t2] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED
2015-08-15 11:42:14,018 DEBUG [AM.ZK.Worker-pool2-t3] master.AssignmentManager: Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1439610133995, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:14,019 INFO [AM.ZK.Worker-pool2-t3] master.RegionStates: Onlined 1588230740 on bigdate5,60020,1439610126975
2015-08-15 11:42:14,021 INFO [master:bigdate1:60000] master.HMaster: hbase:meta assigned=1, rit=false, location=bigdate5,60020,1439610126975
2015-08-15 11:42:14,238 INFO [master:bigdate1:60000] catalog.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-08-15 11:42:14,238 INFO [master:bigdate1:60000] catalog.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-08-15 11:42:14,280 DEBUG [master:bigdate1:60000] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Deleting any existing unassigned nodes
2015-08-15 11:42:14,284 INFO [master:bigdate1:60000] master.AssignmentManager: Clean cluster startup. Assigning user regions
2015-08-15 11:42:14,287 INFO [master:bigdate1:60000] master.SnapshotOfRegionAssignmentFromMeta: Start to scan the hbase:meta for the current region assignment snappshot
2015-08-15 11:42:14,303 INFO [master:bigdate1:60000] master.SnapshotOfRegionAssignmentFromMeta: Finished to scan the hbase:meta for the current region assignmentsnapshot
2015-08-15 11:42:14,313 INFO [master:bigdate1:60000] master.AssignmentManager: Joined the cluster in 75ms, failover=false
2015-08-15 11:42:14,353 INFO [master:bigdate1:60000] master.TableNamespaceManager: Namespace table not found. Creating...
2015-08-15 11:42:14,429 DEBUG [master:bigdate1:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000018
2015-08-15 11:42:14,450 DEBUG [master:bigdate1:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000018
2015-08-15 11:42:14,450 FATAL [master:bigdate1:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-08-15 11:42:14,451 FATAL [master:bigdate1:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1167)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:1015)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:684)
at java.lang.Thread.run(Thread.java:745)
2015-08-15 11:42:14,455 INFO [master:bigdate1:60000] master.HMaster: Aborting
2015-08-15 11:42:14,455 DEBUG [master:bigdate1:60000] master.HMaster: Stopping service threads
2015-08-15 11:42:14,456 INFO [CatalogJanitor-bigdate1:60000] master.CatalogJanitor: CatalogJanitor-bigdate1:60000 exiting
2015-08-15 11:42:14,456 INFO [master:bigdate1:60000] ipc.RpcServer: Stopping server on 60000
2015-08-15 11:42:14,457 INFO [bigdate1,60000,1439610118161-ClusterStatusChore] balancer.ClusterStatusChore: bigdate1,60000,1439610118161-ClusterStatusChore exiting
2015-08-15 11:42:14,456 INFO [bigdate1,60000,1439610118161-BalancerChore] balancer.BalancerChore: bigdate1,60000,1439610118161-BalancerChore exiting
2015-08-15 11:42:14,460 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-08-15 11:42:14,463 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-08-15 11:42:14,464 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-08-15 11:42:14,467 INFO [master:bigdate1:60000.oldLogCleaner] cleaner.LogCleaner: master:bigdate1:60000.oldLogCleaner exiting
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x24f2f58d028000d, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:bigdate1:60000.archivedHFileCleaner exiting
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000] master.HMaster: Stopping infoServer
2015-08-15 11:42:14,475 INFO [master:bigdate1:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x24f2f58d028000d closed
2015-08-15 11:42:14,475 INFO [master:bigdate1:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,477 INFO [master:bigdate1:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2015-08-15 11:42:14,593 DEBUG [master:bigdate1:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@39f52df3
2015-08-15 11:42:14,593 INFO [master:bigdate1:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24f2f58d028000c
2015-08-15 11:42:14,599 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Session: 0x24f2f58d028000c closed
2015-08-15 11:42:14,599 INFO [master:bigdate1:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,702 INFO [bigdate1,60000,1439610118161.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: bigdate1,60000,1439610118161.splitLogManagerTimeoutMonitor exiting
2015-08-15 11:42:14,709 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Session: 0x24f2f58d0280005 closed
2015-08-15 11:42:14,709 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,709 INFO [master:bigdate1:60000] master.HMaster: HMaster main thread exiting
2015-08-15 11:42:14,710 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3031)
本人是新手，32位hadoop编译成64位后，32位hadoop替换成64位的版本。只替换64位中的ETC文件夹。
出现master 闪退现象 回退版本后也无法启动master.用以下方法解决
删除name/ 、data/ 、var/lib/zookeep/
2015-08-15 11:42:07,438 DEBUG [master:bigdate1:60000-WAL.AsyncWriter] wal.FSHLog: master:bigdate1:60000-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer
2015-08-15 11:42:07,438 INFO [master:bigdate1:60000-WAL.AsyncWriter] wal.FSHLog: master:bigdate1:60000-WAL.AsyncWriter exiting
2015-08-15 11:42:07,438 DEBUG [master:bigdate1:60000] wal.FSHLog: Closing WAL writer in hdfs://bigdate1:8020/hbase/data/hbase/meta/1588230740/WALs
2015-08-15 11:42:07,558 DEBUG [master:bigdate1:60000] wal.FSHLog: Moved 1 WAL file(s) to /hbase/data/hbase/meta/1588230740/oldWALs
2015-08-15 11:42:07,769 DEBUG [master:bigdate1:60000] util.FSTableDescriptors: Wrote descriptor into: hdfs://bigdate1:8020/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2015-08-15 11:42:07,898 INFO [master:bigdate1:60000] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2015-08-15 11:42:07,927 DEBUG [master:bigdate1:60000] master.SplitLogManager: Distributed log replay=false, hfile.format.version=2
2015-08-15 11:42:07,936 INFO [master:bigdate1:60000] master.SplitLogManager: Timeout=120000, unassigned timeout=180000, distributedLogReplay=false
2015-08-15 11:42:07,941 INFO [master:bigdate1:60000] master.SplitLogManager: Found 0 orphan tasks and 0 rescan nodes
2015-08-15 11:42:07,942 DEBUG [master:bigdate1:60000] util.FSTableDescriptors: Fetching table descriptors from the filesystem.
2015-08-15 11:42:08,017 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x700a7ec connecting to ZooKeeper ensemble=bigdate5:2181,bigdate4:2181,bigdate6:2181
2015-08-15 11:42:08,017 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=bigdate5:2181,bigdate4:2181,bigdate6:2181 sessionTimeout=90000 watcher=hconnection-0x700a7ec, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:08,019 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Opening socket connection to server bigdate5/172.19.21.105:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-15 11:42:08,020 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Socket connection established to bigdate5/172.19.21.105:2181, initiating session
2015-08-15 11:42:08,034 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Session establishment complete on server bigdate5/172.19.21.105:2181, sessionid = 0x24f2f58d028000c, negotiated timeout = 40000
2015-08-15 11:42:08,092 DEBUG [master:bigdate1:60000] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@39f52df3
2015-08-15 11:42:08,214 INFO [master:bigdate1:60000] master.HMaster: Server active/primary master=bigdate1,60000,1439610118161, sessionid=0x24f2f58d0280005, setting cluster-up flag (Was=true)
2015-08-15 11:42:08,249 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Node /hbase/online-snapshot/acquired already exists and this is not a retry
2015-08-15 11:42:08,252 INFO [master:bigdate1:60000] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort
2015-08-15 11:42:08,256 DEBUG [master:bigdate1:60000] procedure.ZKProcedureCoordinatorRpcs: Starting the controller for procedure member:bigdate1,60000,1439610118161
2015-08-15 11:42:08,310 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_OPEN_REGION-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,310 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_CLOSE_REGION-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_SERVER_OPERATIONS-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_META_SERVER_OPERATIONS-bigdate1:60000, corePoolSize=5, maxPoolSize=5
2015-08-15 11:42:08,311 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=M_LOG_REPLAY_OPS-bigdate1:60000, corePoolSize=10, maxPoolSize=10
2015-08-15 11:42:08,312 DEBUG [master:bigdate1:60000] executor.ExecutorService: Starting executor service name=MASTER_TABLE_OPERATIONS-bigdate1:60000, corePoolSize=1, maxPoolSize=1
2015-08-15 11:42:08,315 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner
2015-08-15 11:42:08,321 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=bigdate5:2181,bigdate4:2181,bigdate6:2181
2015-08-15 11:42:08,321 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=bigdate5:2181,bigdate4:2181,bigdate6:2181 sessionTimeout=90000 watcher=replicationLogCleaner, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:08,323 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Opening socket connection to server bigdate5/172.19.21.105:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-15 11:42:08,327 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Socket connection established to bigdate5/172.19.21.105:2181, initiating session
2015-08-15 11:42:08,333 INFO [master:bigdate1:60000-SendThread(bigdate5:2181)] zookeeper.ClientCnxn: Session establishment complete on server bigdate5/172.19.21.105:2181, sessionid = 0x24f2f58d028000d, negotiated timeout = 40000
2015-08-15 11:42:08,343 INFO [master:bigdate1:60000] zookeeper.RecoverableZooKeeper: Node /hbase/replication/rs already exists and this is not a retry
2015-08-15 11:42:08,343 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
2015-08-15 11:42:08,349 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner
2015-08-15 11:42:08,353 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner
2015-08-15 11:42:08,355 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner
2015-08-15 11:42:08,357 DEBUG [master:bigdate1:60000] cleaner.CleanerChore: initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner
2015-08-15 11:42:08,357 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:08,792 INFO [FifoRpcScheduler.handler1-thread-4] master.ServerManager: Registering server=bigdate4,60020,1439610125637
2015-08-15 11:42:08,796 INFO [FifoRpcScheduler.handler1-thread-4] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-08-15 11:42:08,825 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 468 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:08,843 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,654 INFO [FifoRpcScheduler.handler1-thread-5] master.ServerManager: Registering server=bigdate2,60020,1439610123562
2015-08-15 11:42:09,657 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 2, slept for 1300 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:09,679 INFO [FifoRpcScheduler.handler1-thread-6] master.ServerManager: Registering server=bigdate3,60020,1439610125094
2015-08-15 11:42:09,687 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,688 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:09,709 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 1352 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:09,739 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:09,740 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:09,741 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:10,601 INFO [FifoRpcScheduler.handler1-thread-8] master.ServerManager: Registering server=bigdate1,60020,1439610119732
2015-08-15 11:42:10,634 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:10,636 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:10,638 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:10,640 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:10,645 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 4, slept for 2288 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:11,235 INFO [FifoRpcScheduler.handler1-thread-10] master.ServerManager: Registering server=bigdate5,60020,1439610126975
2015-08-15 11:42:11,269 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 5, slept for 2912 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:11,296 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:11,298 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:11,300 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:11,302 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:11,304 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate5,60020,1439610126975
2015-08-15 11:42:12,778 INFO [master:bigdate1:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 5, slept for 4421 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-08-15 11:42:12,882 INFO [master:bigdate1:60000] master.ServerManager: Finished waiting for region servers count to settle; checked in 5, slept for 4525 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-08-15 11:42:12,891 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate1,60020,1439610119732 belongs to an existing region server
2015-08-15 11:42:12,891 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate2,60020,1439610123562 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate3,60020,1439610125094 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate4,60020,1439610125637 belongs to an existing region server
2015-08-15 11:42:12,892 INFO [master:bigdate1:60000] master.MasterFileSystem: Log folder hdfs://bigdate1:8020/hbase/WALs/bigdate5,60020,1439610126975 belongs to an existing region server
2015-08-15 11:42:13,105 INFO [master:bigdate1:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=bigdate2,60020,1439559068496, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on bigdate2,60020,1439610123562
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3680)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92)
at java.lang.Thread.run(Thread.java:745)
2015-08-15 11:42:13,110 INFO [master:bigdate1:60000] master.MasterFileSystem: Log dir for server bigdate2,60020,1439559068496 does not exist
2015-08-15 11:42:13,110 INFO [master:bigdate1:60000] master.SplitLogManager: dead splitlog workers [bigdate2,60020,1439559068496]
2015-08-15 11:42:13,111 DEBUG [master:bigdate1:60000] master.SplitLogManager: Scheduling batch of logs to split
2015-08-15 11:42:13,112 INFO [master:bigdate1:60000] master.SplitLogManager: started splitting 0 logs in []
2015-08-15 11:42:13,113 INFO [master:bigdate1:60000] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [] in 1ms
2015-08-15 11:42:13,114 INFO [master:bigdate1:60000] zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in ZooKeeper
2015-08-15 11:42:13,129 DEBUG [master:bigdate1:60000] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=bigdate5,60020,1439610126975; 5 (online=5, available=5) available servers, forceNewPlan=false
2015-08-15 11:42:13,129 DEBUG [master:bigdate1:60000] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state
2015-08-15 11:42:13,153 INFO [master:bigdate1:60000] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to bigdate5,60020,1439610126975
2015-08-15 11:42:13,153 INFO [master:bigdate1:60000] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1439610133129, server=null} to {1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,153 DEBUG [master:bigdate1:60000] master.ServerManager: New admin connection to bigdate5,60020,1439610126975
2015-08-15 11:42:13,350 INFO [master:bigdate1:60000] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-08-15 11:42:13,413 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager: Handling RS_ZK_REGION_OPENING, server=bigdate5,60020,1439610126975, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,416 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1439610133153, server=bigdate5,60020,1439610126975} to {1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,537 INFO [FifoRpcScheduler.handler1-thread-15] master.ServerManager: Registering server=bigdate6,60020,1439610131812
2015-08-15 11:42:13,578 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate4,60020,1439610125637
2015-08-15 11:42:13,584 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate1,60020,1439610119732
2015-08-15 11:42:13,586 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate3,60020,1439610125094
2015-08-15 11:42:13,588 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate2,60020,1439610123562
2015-08-15 11:42:13,590 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate6,60020,1439610131812
2015-08-15 11:42:13,592 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/bigdate5,60020,1439610126975
2015-08-15 11:42:13,994 DEBUG [AM.ZK.Worker-pool2-t2] master.AssignmentManager: Handling RS_ZK_REGION_OPENED, server=bigdate5,60020,1439610126975, region=1588230740, current_state={1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:13,995 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1439610133416, server=bigdate5,60020,1439610126975} to {1588230740 state=OPEN, ts=1439610133995, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:14,000 INFO [AM.ZK.Worker-pool2-t2] handler.OpenedRegionHandler: Handling OPENED of 1588230740 from bigdate5,60020,1439610126975; deleting unassigned node
2015-08-15 11:42:14,011 DEBUG [AM.ZK.Worker-pool2-t2] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED
2015-08-15 11:42:14,018 DEBUG [AM.ZK.Worker-pool2-t3] master.AssignmentManager: Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1439610133995, server=bigdate5,60020,1439610126975}
2015-08-15 11:42:14,019 INFO [AM.ZK.Worker-pool2-t3] master.RegionStates: Onlined 1588230740 on bigdate5,60020,1439610126975
2015-08-15 11:42:14,021 INFO [master:bigdate1:60000] master.HMaster: hbase:meta assigned=1, rit=false, location=bigdate5,60020,1439610126975
2015-08-15 11:42:14,238 INFO [master:bigdate1:60000] catalog.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-08-15 11:42:14,238 INFO [master:bigdate1:60000] catalog.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-08-15 11:42:14,280 DEBUG [master:bigdate1:60000] zookeeper.ZKAssign: master:60000-0x24f2f58d0280005, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase Deleting any existing unassigned nodes
2015-08-15 11:42:14,284 INFO [master:bigdate1:60000] master.AssignmentManager: Clean cluster startup. Assigning user regions
2015-08-15 11:42:14,287 INFO [master:bigdate1:60000] master.SnapshotOfRegionAssignmentFromMeta: Start to scan the hbase:meta for the current region assignment snappshot
2015-08-15 11:42:14,303 INFO [master:bigdate1:60000] master.SnapshotOfRegionAssignmentFromMeta: Finished to scan the hbase:meta for the current region assignmentsnapshot
2015-08-15 11:42:14,313 INFO [master:bigdate1:60000] master.AssignmentManager: Joined the cluster in 75ms, failover=false
2015-08-15 11:42:14,353 INFO [master:bigdate1:60000] master.TableNamespaceManager: Namespace table not found. Creating...
2015-08-15 11:42:14,429 DEBUG [master:bigdate1:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000018
2015-08-15 11:42:14,450 DEBUG [master:bigdate1:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000018
2015-08-15 11:42:14,450 FATAL [master:bigdate1:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-08-15 11:42:14,451 FATAL [master:bigdate1:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1167)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:1015)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:684)
at java.lang.Thread.run(Thread.java:745)
2015-08-15 11:42:14,455 INFO [master:bigdate1:60000] master.HMaster: Aborting
2015-08-15 11:42:14,455 DEBUG [master:bigdate1:60000] master.HMaster: Stopping service threads
2015-08-15 11:42:14,456 INFO [CatalogJanitor-bigdate1:60000] master.CatalogJanitor: CatalogJanitor-bigdate1:60000 exiting
2015-08-15 11:42:14,456 INFO [master:bigdate1:60000] ipc.RpcServer: Stopping server on 60000
2015-08-15 11:42:14,457 INFO [bigdate1,60000,1439610118161-ClusterStatusChore] balancer.ClusterStatusChore: bigdate1,60000,1439610118161-ClusterStatusChore exiting
2015-08-15 11:42:14,456 INFO [bigdate1,60000,1439610118161-BalancerChore] balancer.BalancerChore: bigdate1,60000,1439610118161-BalancerChore exiting
2015-08-15 11:42:14,460 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-08-15 11:42:14,463 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-08-15 11:42:14,464 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-08-15 11:42:14,467 INFO [master:bigdate1:60000.oldLogCleaner] cleaner.LogCleaner: master:bigdate1:60000.oldLogCleaner exiting
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x24f2f58d028000d, quorum=bigdate5:2181,bigdate4:2181,bigdate6:2181, baseZNode=/hbase
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:bigdate1:60000.archivedHFileCleaner exiting
2015-08-15 11:42:14,468 INFO [master:bigdate1:60000] master.HMaster: Stopping infoServer
2015-08-15 11:42:14,475 INFO [master:bigdate1:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x24f2f58d028000d closed
2015-08-15 11:42:14,475 INFO [master:bigdate1:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,477 INFO [master:bigdate1:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2015-08-15 11:42:14,593 DEBUG [master:bigdate1:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@39f52df3
2015-08-15 11:42:14,593 INFO [master:bigdate1:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24f2f58d028000c
2015-08-15 11:42:14,599 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Session: 0x24f2f58d028000c closed
2015-08-15 11:42:14,599 INFO [master:bigdate1:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,702 INFO [bigdate1,60000,1439610118161.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: bigdate1,60000,1439610118161.splitLogManagerTimeoutMonitor exiting
2015-08-15 11:42:14,709 INFO [master:bigdate1:60000] zookeeper.ZooKeeper: Session: 0x24f2f58d0280005 closed
2015-08-15 11:42:14,709 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-08-15 11:42:14,709 INFO [master:bigdate1:60000] master.HMaster: HMaster main thread exiting
2015-08-15 11:42:14,710 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3031)
本人是新手，32位hadoop编译成64位后，32位hadoop替换成64位的版本。只替换64位中的ETC文件夹。
出现master 闪退现象 回退版本后也无法启动master.用以下方法解决
删除name/ 、data/ 、var/lib/zookeep/
/2 
