hbase regionserver exception

早上起来发现hbase有几个节点down机了，查看日志发现每台机器的报错日志都一样：

2013-12-28 10:40:01,702 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at master,60000,1388136711301
2013-12-28 10:40:08,769 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.03 MB, free=244.85 MB, max=246.88 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cach
ingHitsRatio=0, evictions=0, evicted=0, evictedPerRun=NaN
2013-12-28 10:40:11,714 INFO org.apache.hadoop.ipc.HBaseRPC: Problem connecting to server: master/192.168.1.181:60000
2013-12-28 10:40:11,715 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.SocketTimeoutException: Call to master/192.168.1.181:60000 failed on socket timeout exception: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for read. ch : j
ava.nio.channels.SocketChannel[connected local=/192.168.1.179:51454 remote=master/192.168.1.181:60000]
      at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1043)
      at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1016)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
      at com.sun.proxy.$Proxy9.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:141)
      at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2036)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2082)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:751)
      at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.1.179:51454 remote=master/192.168.1.18
1:60000]
      at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
      at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
      at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
      at java.io.FilterInputStream.read(FilterInputStream.java:116)
      at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:374)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
      at java.io.DataInputStream.readInt(DataInputStream.java:370)
      at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:654)
      at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:588)