求教 提交了任务java.io.IOException: Broken pipe

但是 大部分的map都是fail了，只有个别3、4个map可以成功
找了下失败的map的日志，如下
2015-07-14 15:52:03,699 INFO [main] org.apache.hadoop.streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
2015-07-14 16:02:19,579 INFO [main] org.apache.hadoop.streaming.PipeMapRed: R/W/S=214/0/0 in:0=214/615 [rec/s] out:0=0/615 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 HOST=null
USER=work
HADOOP_USER=null
last tool output: |null|

java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:282)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.streaming.io.TextInputWriter.writeUTF8(TextInputWriter.java:72)
        at org.apache.hadoop.streaming.io.TextInputWriter.writeValue(TextInputWriter.java:51)
        at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:106)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
2015-07-14 16:02:19,577 INFO [Thread-17] org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
2015-07-14 16:02:19,581 WARN [main] org.apache.hadoop.streaming.PipeMapRed: java.io.IOException: Broken pipe
2015-07-14 16:02:19,583 INFO [main] org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 143

分享解决办法：


查看dfs.data.dir属性所指的路径是否磁盘已经满了，如果满了则进行处理后再次尝试hadoop fs -put数据。
如果相关磁盘没有满，则需要排查相关磁盘没有坏扇区，需要检测。