本帖最后由 redhat1986 于 2016-11-1 20:17 编辑
2016-11-01 01:51:39,511 WARN [s3sa114,16000,1477439159914_ChoreService_2] master.HMaster: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure Unable to list backup servers
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/backup-masters
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
2016-11-01 01:55:02,562 INFO [s3sa114,16000,1477439159914_ChoreService_1] cleaner.HFileCleaner: Chore: HFileCleaner missed its start time
2016-11-01 05:18:36,424 INFO [s3sa114,16000,1477439159914_ChoreService_1] retry.RetryInvocationHandler: Exception while invoking getListing of class ClientNamenodeProtocolTranslatorPB over s3sa112/10.58.72.1:8020 after 10 fail over att
empts. Trying to fail over after sleeping for 15513ms.
2016-11-01 05:18:37,646 WARN [s3sa114,16000,1477439159914_ChoreService_2] retry.RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.
2016-11-01 05:18:38,086 WARN [S3SA114:16000.activeMasterManager-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Session 0x357fc175a680268 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.NoRouteToHostException: No route to host
2016-11-01 05:18:38,591 WARN [S3SA114:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone need
s it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:610)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2016-11-01 05:18:38,591 INFO [S3SA114:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x357fc175a680266
2016-11-01 05:18:38,591 INFO [S3SA114:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-11-01 05:18:38,861 INFO [main-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA116/10.58.72.24:2181, initiating session
2016-11-01 05:18:38,861 INFO [main-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x157fbe0dd6b03b7 has expired, closing socket connection
2016-11-01 05:18:38,862 FATAL [main-EventThread] master.HMaster: Master server abort: loaded coprocessors are: []
2016-11-01 05:18:38,862 FATAL [main-EventThread] master.HMaster: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure master:16000-0x157fbe0dd6b03b7 received expired from ZooKeeper, ab
orting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:610)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2016-11-01 05:18:38,862 INFO [main-EventThread] regionserver.HRegionServer: STOPPED: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure master:16000-0x157fbe0dd6b03b7 received expir
ed from ZooKeeper, aborting
2016-11-01 05:18:38,862 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-11-01 05:18:38,862 INFO [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: Stopping infoServer
2016-11-01 05:18:38,864 INFO [master/S3SA114/10.58.72.3:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2016-11-01 05:18:38,914 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Opening socket connection to server S3SA119/10.58.72.26:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-01 05:18:38,914 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA119/10.58.72.26:2181, initiating session
2016-11-01 05:18:38,915 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x157fbe0dd6b1977, likely server has closed socket, closing socket connect
ion and attempting reconnect
2016-11-01 05:18:38,923 INFO [s3sa114,16000,1477439159914_splitLogManager__ChoreService_1] master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout Monitor was stopped
2016-11-01 05:18:38,962 WARN [s3sa114,16000,1477439159914_ChoreService_4] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$Sessi
onExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/backup-masters
2016-11-01 05:18:38,966 INFO [master/S3SA114/10.58.72.3:16000] procedure2.ProcedureExecutor: Stopping the procedure executor
2016-11-01 05:18:38,966 INFO [master/S3SA114/10.58.72.3:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
2016-11-01 05:18:39,161 INFO [S3SA114:16000.activeMasterManager-SendThread(S3SA117:2181)] zookeeper.ClientCnxn: Opening socket connection to server S3SA117/10.58.72.25:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-01 05:18:39,161 INFO [S3SA114:16000.activeMasterManager-SendThread(S3SA117:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA117/10.58.72.25:2181, initiating session
2016-11-01 05:18:39,162 INFO [S3SA114:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-11-01 05:18:39,162 INFO [S3SA114:16000.activeMasterManager-SendThread(S3SA117:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x357fc175a680268 has expired, closing socket connection
2016-11-01 05:18:39,959 WARN [s3sa114,16000,1477439159914_ChoreService_2] cleaner.CleanerChore: A file cleanerLogsCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/oldWALs
2016-11-01 05:18:39,960 INFO [s3sa114,16000,1477439159914_ChoreService_2] cleaner.LogCleaner: Chore: LogsCleaner missed its start time
2016-11-01 05:18:39,960 INFO [s3sa114,16000,1477439159914_ChoreService_3] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x357fc175a680268, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure
2016-11-01 05:18:39,960 INFO [s3sa114,16000,1477439159914_ChoreService_3] cleaner.LogCleaner: Chore: LogsCleaner was stopped
2016-11-01 05:18:39,976 WARN [snapshot-hfile-cleaner-cache-refresher] retry.RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.
2016-11-01 05:18:40,169 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Opening socket connection to server S3SA119/10.58.72.26:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-01 05:18:40,169 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA119/10.58.72.26:2181, initiating session
2016-11-01 05:18:40,170 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA119:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x357fc175a680265, likely server has closed socket, closing socket conne
ction and attempting reconnect
2016-11-01 05:18:40,473 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Opening socket connection to server S3SA116/10.58.72.24:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-01 05:18:40,473 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA116/10.58.72.24:2181, initiating session
2016-11-01 05:18:40,474 WARN [2070393899@qtp-958154311-1048-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:610)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2016-11-01 05:18:40,474 INFO [2070393899@qtp-958154311-1048-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x157fbe0dd6b1977 has expired, closing socket connection
2016-11-01 05:18:40,475 INFO [2070393899@qtp-958154311-1048-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x157fbe0dd6b1977
2016-11-01 05:18:40,475 INFO [2070393899@qtp-958154311-1048-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-11-01 05:18:41,138 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Opening socket connection to server S3SA116/10.58.72.24:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-01 05:18:41,138 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Socket connection established to S3SA116/10.58.72.24:2181, initiating session
2016-11-01 05:18:41,139 INFO [master/S3SA114/10.58.72.3:16000-SendThread(S3SA116:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x357fc175a680265 has expired, closing socket connection
2016-11-01 05:18:41,139 WARN [master/S3SA114/10.58.72.3:16000-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs
it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:610)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2016-11-01 05:18:41,140 INFO [master/S3SA114/10.58.72.3:16000-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x357fc175a680265
2016-11-01 05:18:41,140 INFO [master/S3SA114/10.58.72.3:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-11-01 05:18:46,962 WARN [s3sa114,16000,1477439159914_ChoreService_4] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$Sessi
onExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/backup-masters
2016-11-01 05:18:46,962 ERROR [s3sa114,16000,1477439159914_ChoreService_4] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2016-11-01 05:18:46,962 WARN [s3sa114,16000,1477439159914_ChoreService_4] master.HMaster: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure Unable to list backup servers
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/backup-masters
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:635)
at org.apache.hadoop.hbase.master.HMaster.getClusterStatus(HMaster.java:2135)
at org.apache.hadoop.hbase.master.balancer.ClusterStatusChore.chore(ClusterStatusChore.java:49)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:18:49,178 INFO [WALProcedureStoreSyncThread] retry.RetryInvocationHandler: Exception while invoking create of class ClientNamenodeProtocolTranslatorPB over s3sa113/10.58.72.2:8020 after 6 fail over attempts. Trying to fai
l over after sleeping for 14555ms.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1932)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1313)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:678)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy15.create(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy16.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1720)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
2016-11-01 05:18:51,937 WARN [s3sa114,16000,1477439159914_ChoreService_1] retry.RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.
2016-11-01 05:18:51,941 INFO [s3sa114,16000,1477439159914_ChoreService_1] retry.RetryInvocationHandler: Exception while invoking getListing of class ClientNamenodeProtocolTranslatorPB over s3sa113/10.58.72.2:8020 after 11 fail over att
empts. Trying to fail over after sleeping for 9342ms.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1932)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1313)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4493)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:1063)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:634)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy15.getListing(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy16.getListing(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.getListing(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2094)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2077)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:801)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:863)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:859)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:859)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1721)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1741)
at org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:123)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:01,287 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,287 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,288 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,288 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,289 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,290 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,290 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,291 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,291 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,292 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,292 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,293 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,763 WARN [s3sa114,16000,1477439159914_ChoreService_1] cleaner.CleanerChore: A file cleanerHFileCleaner is stopped, won't delete any more files in:hdfs://gomebigdata-hdfs/apps/hbase/data/archive
2016-11-01 05:19:01,763 INFO [s3sa114,16000,1477439159914_ChoreService_1] cleaner.HFileCleaner: Chore: HFileCleaner missed its start time
2016-11-01 05:19:01,763 INFO [s3sa114,16000,1477439159914_ChoreService_2] cleaner.HFileCleaner: Chore: HFileCleaner was stopped
2016-11-01 05:19:03,734 WARN [WALProcedureStoreSyncThread] retry.RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.
2016-11-01 05:19:03,777 WARN [WALProcedureStoreSyncThread] wal.WALProcedureStore: Unable to write the trailer: Lease timeout of 0 seconds expired.
2016-11-01 05:19:03,777 ERROR [WALProcedureStoreSyncThread] wal.WALProcedureStore: Unable to close the stream
java.io.IOException: Lease timeout of 0 seconds expired.
at org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2250)
at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:949)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:903)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:03,790 INFO [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: stopping server s3sa114,16000,1477439159914
2016-11-01 05:19:03,790 INFO [master/S3SA114/10.58.72.3:16000] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2016-11-01 05:19:03,790 INFO [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: stopping server s3sa114,16000,1477439159914; all regions closed.
2016-11-01 05:19:03,791 INFO [master/S3SA114/10.58.72.3:16000] hbase.ChoreService: Chore service for: s3sa114,16000,1477439159914 had [[ScheduledChore: Name: s3sa114,16000,1477439159914-RegionNormalizerChore Period: 1800000 Unit: MILLI
SECONDS], [ScheduledChore: Name: s3sa114,16000,1477439159914-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: s3sa114,16000,1477439159914-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Nam
e: CatalogJanitor-S3SA114:16000 Period: 300000 Unit: MILLISECONDS]] on shutdown
2016-11-01 05:19:03,791 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/master
2016-11-01 05:19:04,792 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/master
2016-11-01 05:19:06,792 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/master
2016-11-01 05:19:10,792 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/master
2016-11-01 05:19:18,793 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/master
2016-11-01 05:19:18,793 ERROR [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 4 attempts
2016-11-01 05:19:18,793 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.ZKUtil: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure Unable to get data of znode /hbase-unsecure/maste
r
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1164)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1071)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:18,793 ERROR [master/S3SA114/10.58.72.3:16000] zookeeper.ZooKeeperWatcher: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re
-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1164)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1071)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:18,793 ERROR [master/S3SA114/10.58.72.3:16000] master.ActiveMasterManager: master:16000-0x157fbe0dd6b03b7, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, baseZNode=/hbase-unsecure Error deleting our own master address n
ode
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1164)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1071)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:18,793 INFO [master/S3SA114/10.58.72.3:16000] hbase.ChoreService: Chore service for: s3sa114,16000,1477439159914_splitLogManager_ had [] on shutdown
2016-11-01 05:19:18,793 INFO [master/S3SA114/10.58.72.3:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2016-11-01 05:19:18,793 INFO [master/S3SA114/10.58.72.3:16000] ipc.RpcServer: Stopping server on 16000
2016-11-01 05:19:18,793 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2016-11-01 05:19:18,794 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-11-01 05:19:18,794 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2016-11-01 05:19:18,794 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
2016-11-01 05:19:19,794 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
2016-11-01 05:19:21,795 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
2016-11-01 05:19:25,795 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
2016-11-01 05:19:33,795 WARN [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=s3sa117:2181,s3sa119:2181,s3sa116:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredEx
ception: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
2016-11-01 05:19:33,796 ERROR [master/S3SA114/10.58.72.3:16000] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2016-11-01 05:19:33,796 WARN [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/s3sa114,16000,1477439159914
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1345)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1334)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1403)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1079)
at java.lang.Thread.run(Thread.java:745)
2016-11-01 05:19:33,796 INFO [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: stopping server s3sa114,16000,1477439159914; zookeeper connection closed.
2016-11-01 05:19:33,796 INFO [master/S3SA114/10.58.72.3:16000] regionserver.HRegionServer: master/S3SA114/10.58.72.3:16000 exiting
Tue Nov 1 07:19:12 CST 2016 Starting master on S3SA114
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031399
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
2016-11-01 07:19:13,683 INFO [main] util.VersionInfo: HBase 1.1.2.2.3.4.0-3485
2016-11-01 07:19:13,683 INFO [main] util.VersionInfo: Source code repository git://c66-slave-20176e25-2/grid/0/jenkins/workspace/HDP-build-centos6/bigtop/build/hbase/rpm/BUILD/hbase-1.1.2.2.3.4.0 revision=a2b7fd38f4dcca6eb81f99b5d37b2e
a3beeef09e
2016-11-01 07:19:13,683 INFO [main] util.VersionInfo: Compiled by jenkins on Wed Dec 16 03:49:28 UTC 2015
2016-11-01 07:19:13,683 INFO [main] util.VersionInfo: From source with checksum 2272034a01b6bfb7ab2804c1d8be0c67
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/usr/java/jdk7
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/usr/hdp/current/hbase-master/bin/..
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:MAIL=/var/spool/mail/hbase
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:HOSTNAME=S3SA114
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:PWD=/home/hbase
2016-11-01 07:19:14,057 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/var/run/hbase/hbase-hbase-master.znode
2016-11-01 07:19:14,058 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -Xmx24576m -XX:PermSize=128m -XX:MaxPermSize=128m
2016-11-01 07:19:14,058 INFO [main] util.ServerCommandLine: env:CVS_RSH=ssh
2016-11-01 07:19:14,058 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false
|
/2 
京ICP备2020039040号
简书网举报电话：021-34700000
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡
