BlockSender.sendChunks() exception

2019-04-11 15:17:07,016 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception:
java.io.IOException: 断开的管道
      at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
      at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
      at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
      at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
      at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:635)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:819)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:766)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:551)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
      at java.lang.Thread.run(Thread.java:748)
2019-04-11 15:17:07,322 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception:
java.io.IOException: 断开的管道
      at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
      at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
      at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
      at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
      at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:635)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:819)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:766)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:551)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
      at java.lang.Thread.run(Thread.java:748)
2019-04-11 15:17:08,766 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1075676861_1937850 file /home/dfs/dn/current/BP-1811233492-192.168.1.11-1539253579675/current/finalized/subdir29/subdir134/blk_1075676861 for deletion

bioger_hit · 发表于 2019-4-11 18:38:19

异常信息跟 DN 的一个配置有关系，具体配置项是 DN 可以同时处理的文件上限。对于老版本配置项名称为“dfs.datanode.max.xcievers”，对于新版本配置项名称改为“dfs.datanode.max.transfer.threads”。该参数的默认值为 4096，所以需要修改为 8192。