Hregionserver异常退出

当我用sqoop跑数到hbase时，出现hregionserver进程宕掉的情况，查看hbase-regionserver日志，发现报错日志如下，网上百度了没找到解决方案，求大神帮忙解答一下，谢谢！！新人求罩
2015-06-18 12:17:47,917 DEBUG [RS_CLOSE_REGION-master:60020-1] handler.CloseRegionHandler: Closed blog,,1434423637657.506e001e4e95efeaae333d2b4
68dabab.
2015-06-18 12:17:47,917 DEBUG [RS_CLOSE_REGION-master:60020-1] handler.CloseRegionHandler: Processing close of MM_PURCHASE,,1433320491721.de32c
65df48cd093ca438c76f657c43e.
2015-06-18 12:17:47,918 INFO  [StoreCloserThread-hbase:namespace,,1428388421684.9dd2f45c91424a7aa48784a43896b265.-1] regionserver.HStore: Close
d info
2015-06-18 12:17:47,918 INFO  [RS_CLOSE_REGION-master:60020-2] regionserver.HRegion: Closed hbase:namespace,,1428388421684.9dd2f45c91424a7aa487
84a43896b265.
2015-06-18 12:17:47,918 DEBUG [RS_CLOSE_REGION-master:60020-2] handler.CloseRegionHandler: Closed hbase:namespace,,1428388421684.9dd2f45c91424a
7aa48784a43896b265.
2015-06-18 12:17:47,920 DEBUG [RS_CLOSE_REGION-master:60020-1] regionserver.HRegion: Closing MM_PURCHASE,,1433320491721.de32c65df48cd093ca438c7
6f657c43e.: disabling compactions & flushes
2015-06-18 12:17:47,920 DEBUG [RS_CLOSE_REGION-master:60020-1] regionserver.HRegion: Updates disabled for region MM_PURCHASE,,1433320491721.de3
2c65df48cd093ca438c76f657c43e.
2015-06-18 12:17:47,923 INFO  [StoreCloserThread-MM_OPERATION_HISTORY_ARCHIVED,,1433231724664.14462876a0b96e724f5948cbce48245f.-1] regionserver
.HStore: Closed info
2015-06-18 12:17:47,924 INFO  [RS_CLOSE_REGION-master:60020-0] regionserver.HRegion: Closed MM_OPERATION_HISTORY_ARCHIVED,,1433231724664.144628
76a0b96e724f5948cbce48245f.
2015-06-18 12:17:48,791 DEBUG [RS_CLOSE_REGION-master:60020-0] handler.CloseRegionHandler: Closed MM_OPERATION_HISTORY_ARCHIVED,,1433231724664.
14462876a0b96e724f5948cbce48245f.
2015-06-18 12:17:48,798 INFO  [StoreCloserThread-MM_PURCHASE,,1433320491721.de32c65df48cd093ca438c76f657c43e.-1] regionserver.HStore: Closed in
fo
2015-06-18 12:17:48,798 INFO  [RS_CLOSE_REGION-master:60020-1] regionserver.HRegion: Closed MM_PURCHASE,,1433320491721.de32c65df48cd093ca438c76
f657c43e.
2015-06-18 12:17:48,798 DEBUG [RS_CLOSE_REGION-master:60020-1] handler.CloseRegionHandler: Closed MM_PURCHASE,,1433320491721.de32c65df48cd093ca
438c76f657c43e.
2015-06-18 12:17:48,918 INFO  [regionserver60020] regionserver.HRegionServer: stopping server master,60020,1434594391843; all regions closed.
2015-06-18 12:17:49,008 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncNotifier] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncNotifier interru
pted while waiting for  notification from AsyncSyncer thread
2015-06-18 12:17:49,009 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncNotifier] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncNotifier exiting
2015-06-18 12:17:49,154 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer0] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer0 interrupt
ed while waiting for notification from AsyncWriter thread
2015-06-18 12:17:49,154 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer0] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer0 exiting
2015-06-18 12:17:49,198 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer1] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer1 interrupt
ed while waiting for notification from AsyncWriter thread
2015-06-18 12:17:49,198 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer1] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer1 exiting
2015-06-18 12:17:49,228 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer2] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer2 interrupt
ed while waiting for notification from AsyncWriter thread
2015-06-18 12:17:49,228 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer2] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer2 exiting
2015-06-18 12:17:49,247 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer3] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer3 interrupt
ed while waiting for notification from AsyncWriter thread
2015-06-18 12:17:49,247 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer3] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer3 exiting
2015-06-18 12:17:49,341 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer4] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer4 interrupt
ed while waiting for notification from AsyncWriter thread
2015-06-18 12:17:49,341 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncSyncer4] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncSyncer4 exiting
2015-06-18 12:17:49,384 DEBUG [RS_OPEN_META-master:60020-0-WAL.AsyncWriter] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncWriter interrupted
while waiting for newer writes added to local buffer
2015-06-18 12:17:49,385 INFO  [RS_OPEN_META-master:60020-0-WAL.AsyncWriter] wal.FSHLog: RS_OPEN_META-master:60020-0-WAL.AsyncWriter exiting
2015-06-18 12:17:49,385 DEBUG [regionserver60020] wal.FSHLog: Closing WAL writer in hdfs://master:9000/hbase/WALs/master,60020,1434594391843
2015-06-18 12:17:49,429 ERROR [regionserver60020] wal.ProtobufLogWriter: Got IOException while writing trailer
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1744284786-192.168.1.110-1428053526166:blk_1073744859_4199 does not exist or is
not under Constructionblk_1073744859_4326
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:
774)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294)
        at $Proxy15.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1035)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
2015-06-18 12:17:51,448 ERROR [regionserver60020] regionserver.HRegionServer: Metalog close and delete failed
java.io.IOException: BP-1744284786-192.168.1.110-1428053526166:blk_1073744859_4199 does not exist or is not under Constructionblk_1073744859_43
26
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:97)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:49)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:1234)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1033)
        at java.lang.Thread.run(Thread.java:722)
2015-06-18 12:17:51,534 DEBUG [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier interrupted while waiting f
or  notification from AsyncSyncer thread
2015-06-18 12:17:51,534 INFO  [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier exiting
2015-06-18 12:17:51,563 DEBUG [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 interrupted while waiting for
notification from AsyncWriter thread
2015-06-18 12:17:51,564 INFO  [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 exiting
2015-06-18 12:17:51,596 DEBUG [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 interrupted while waiting for
notification from AsyncWriter thread
2015-06-18 12:17:51,596 INFO  [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 exiting
2015-06-18 12:17:51,624 DEBUG [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 interrupted while waiting for
notification from AsyncWriter thread
2015-06-18 12:17:51,624 INFO  [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 exiting
2015-06-18 12:17:51,661 DEBUG [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 interrupted while waiting for
notification from AsyncWriter thread
2015-06-18 12:17:51,661 INFO  [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 exiting
2015-06-18 12:17:51,661 DEBUG [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 interrupted while waiting for
notification from AsyncWriter thread
2015-06-18 12:17:51,661 INFO  [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 exiting
2015-06-18 12:17:51,689 DEBUG [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter interrupted while waiting for n
ewer writes added to local buffer
2015-06-18 12:17:51,689 INFO  [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter exiting
2015-06-18 12:17:51,689 DEBUG [regionserver60020] wal.FSHLog: Closing WAL writer in hdfs://master:9000/hbase/WALs/master,60020,1434594391843
2015-06-18 12:17:51,689 ERROR [regionserver60020] wal.ProtobufLogWriter: Got IOException while writing trailer
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1744284786-192.168.1.110-1428053526166:blk_1073744858_4198 does not exist or is
not under Constructionblk_1073744858_4198{blockUCState=UNDER_RECOVERY, primaryNodeIndex=0, replicas=[ReplicaUnderConstruction[[DISK]DS-55e00eeb
-a9f3-4745-905b-c0d1abf626b7:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-5b6cf811-6f7a-44b4-b25c-80b969581952:NORMAL|RBW]]}
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:
774)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294)
        at $Proxy15.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1035)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
2015-06-18 12:17:51,690 ERROR [regionserver60020] regionserver.HRegionServer: Close and delete failed
java.io.IOException: BP-1744284786-192.168.1.110-1428053526166:blk_1073744858_4198 does not exist or is not under Constructionblk_1073744858_41
98{blockUCState=UNDER_RECOVERY, primaryNodeIndex=0, replicas=[ReplicaUnderConstruction[[DISK]DS-55e00eeb-a9f3-4745-905b-c0d1abf626b7:NORMAL|RBW
], ReplicaUnderConstruction[[DISK]DS-5b6cf811-6f7a-44b4-b25c-80b969581952:NORMAL|RBW]]}
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:97)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:49)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:1245)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1033)
        at java.lang.Thread.run(Thread.java:722)
2015-06-18 12:17:51,690 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closing leases
2015-06-18 12:17:51,695 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closed leases
2015-06-18 12:17:51,781 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/master,600
20,1434594391843
2015-06-18 12:17:52,021 INFO  [regionserver60020] util.RetryCounter: Sleeping 1000ms before retry #0...
2015-06-18 12:17:53,022 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/master,600
20,1434594391843
2015-06-18 12:17:53,022 INFO  [regionserver60020] util.RetryCounter: Sleeping 2000ms before retry #1...
2015-06-18 12:17:55,023 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/master,600
20,1434594391843
2015-06-18 12:17:55,023 INFO  [regionserver60020] util.RetryCounter: Sleeping 4000ms before retry #2...
2015-06-18 12:17:59,024 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/master,600
20,1434594391843
2015-06-18 12:17:59,060 INFO  [regionserver60020] util.RetryCounter: Sleeping 8000ms before retry #3...
2015-06-18 12:18:07,061 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/master,600
20,1434594391843
2015-06-18 12:18:07,061 ERROR [regionserver60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2015-06-18 12:18:08,603 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,143459439
1843
2015-06-18 12:18:08,603 INFO  [regionserver60020] util.RetryCounter: Sleeping 1000ms before retry #0...
2015-06-18 12:18:09,604 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,143459439
1843
2015-06-18 12:18:09,605 INFO  [regionserver60020] util.RetryCounter: Sleeping 2000ms before retry #1...
2015-06-18 12:18:11,606 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,143459439
1843
2015-06-18 12:18:11,607 INFO  [regionserver60020] util.RetryCounter: Sleeping 4000ms before retry #2...
2015-06-18 12:18:15,607 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,143459439
1843
2015-06-18 12:18:15,607 INFO  [regionserver60020] util.RetryCounter: Sleeping 8000ms before retry #3...
2015-06-18 12:18:23,608 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave1:2181,master:2181,
exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,143459439
1843
2015-06-18 12:18:23,608 ERROR [regionserver60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2015-06-18 12:18:23,608 WARN  [regionserver60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/master,60020,1434594391843
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:179)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1273)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1262)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1343)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1055)
        at java.lang.Thread.run(Thread.java:722)
2015-06-18 12:18:23,672 INFO  [regionserver60020] regionserver.HRegionServer: stopping server master,60020,1434594391843; zookeeper connection
closed.
2015-06-18 12:18:23,672 INFO  [regionserver60020] regionserver.HRegionServer: regionserver60020 exiting
2015-06-18 12:18:23,735 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2507)
2015-06-18 12:18:24,432 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache
.hadoop.fs.FileSystem$Cache$ClientFinalizer@32cc9262
2015-06-18 12:18:24,439 INFO  [Thread-9] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2015-06-18 12:18:24,456 ERROR [Thread-309] hdfs.DFSClient: Failed to close file /hbase/WALs/master,60020,1434594391843/master%2C60020%2C1434594
391843.1434598008433
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1744284786-192.168.1.110-1428053526166:blk_1073744858_4198 does not exist or is
not under Constructionblk_1073744858_4198{blockUCState=UNDER_RECOVERY, primaryNodeIndex=0, replicas=[ReplicaUnderConstruction[[DISK]DS-55e00eeb
-a9f3-4745-905b-c0d1abf626b7:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-5b6cf811-6f7a-44b4-b25c-80b969581952:NORMAL|RBW]]}
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:
774)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294)
        at $Proxy15.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1035)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
2015-06-18 12:18:24,456 ERROR [Thread-309] hdfs.DFSClient: Failed to close file /hbase/WALs/master,60020,1434594391843/master%2C60020%2C1434594
391843.1434598008462.meta
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1744284786-192.168.1.110-1428053526166:blk_1073744859_4199 does not exist or is
not under Constructionblk_1073744859_4326
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6273)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:682)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSi
deTranslatorPB.java:889)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtoco
lProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at $Proxy14.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:
774)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294)
        at $Proxy15.updateBlockForPipeline(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1035)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
2015-06-18 12:18:24,497 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook finished.

楼主这个问题很多，首先确保hadoop各个进程都是存在的，确保不存在僵死进程。
datanode能够写数据

看着像是hadoop和zookeeper协议都对不上，启动的版本是不是有问题？
另外可以看看zookeer状态是否正常，是否有部分zookeepr节点挂掉了。

楼主问题解决了吗，是不是zookeeper的问题

BP-1744284786-192.168.1.110-1428053526166:blk_1073744859_4199 does not exist or is
not under Constructionblk_1073744859_4326
检查一下你的hdfs，看看是不是有文件丢失 ，或者有硬件故障！！


另外用hbase hbck检查修复一下！  metal表里面的信息对不上了，重新修复hbase meta表！