Hentai 发表于 2016-12-27 09:24 hdfs.sinks.fs.hdfs.rollInterval=21600 hdfs.sinks.fs.hdfs.rollSize=8589934592 为何这两个设置这么大。超时出现问题正常的设置小一点，根据情况自己调试下。 建议初始值： hdfs.sinks.fs.hdfs.rollInterval=10 hdfs.sinks.fs.hdfs.rollSize=0 依次： 20 40 80 200 等值。 |
langke93 发表于 2016-12-26 19:24 配置 hdfs.sinks.fs.channel=fc hdfs.sources.fr.channels=fc hdfs.sources.fr.type=exec hdfs.sources.fr.command=tshark -i em2 -t ad hdfs.sinks.fs.type=hdfs hdfs.sinks.fs.hdfs.path=hdfs://10.0.40.2:8020/user/xiakang #hdfs.sinks.fs.hdfs.filePrefix=Syslog hdfs.sinks.fs.hdfs.rollInterval=21600 hdfs.sinks.fs.hdfs.rollSize=8589934592 hdfs.sinks.fs.hdfs.rollCount=0 hdfs.sinks.fs.hdfs.fileType=DataStream hdfs.sinks.fs.hdfs.minBlockReplicas=1 hdfs.sinks.fs.hdfs.inUseSuffix=log hdfs.channels.fc.type=SPILLABLEMEMORY hdfs.channels.fc.checkpointDir = /mnt/flume/checkpoint hdfs.channels.fc.dataDirs = /mnt/flume/data 版本 Flume 1.5.2.2.5.0.0-1245 错误 Failed while running command: tshark -i em2 -t ad java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) at java.io.BufferedInputStream.read1(BufferedInputStream.java:283) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:331) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 26 Dec 2016 16:23:41,039 INFO [pool-5-thread-1] (org.apache.flume.source.ExecSource$ExecRu nnable.run:370) - Command [tshark -i em2 -t ad] exited with 141 26 Dec 2016 16:23:41,040 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:149) - Component type: SOURCE, name: fr stopped 26 Dec 2016 16:23:41,040 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:155) - Shutdown Metric for type: SOURCE, name: fr. source.start.tim e == 1482740567800 26 Dec 2016 16:23:41,040 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:161) - Shutdown Metric for type: SOURCE, name: fr. source.stop.time == 1482740621040 26 Dec 2016 16:23:41,041 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.append-batch .accepted == 0 26 Dec 2016 16:23:41,041 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.append-batch .received == 0 26 Dec 2016 16:23:41,041 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.append.accep ted == 0 26 Dec 2016 16:23:41,041 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.append.recei ved == 0 26 Dec 2016 16:23:41,041 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.events.accep ted == 514320 26 Dec 2016 16:23:41,042 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.events.recei ved == 514334 26 Dec 2016 16:23:41,042 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni toredCounterGroup.stop:177) - Shutdown Metric for type: SOURCE, name: fr. src.open-connect ion.count == 0 26 Dec 2016 16:23:41,042 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache. flume.sink.hdfs.HDFSEventSink.process:466) - process failed java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket Writer.java:687) at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) 26 Dec 2016 16:23:41,043 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache. flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before H DFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:470) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.c allTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket Writer.java:687) at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448) ... 3 more |
Hentai 发表于 2016-12-26 16:46 没遇到过，楼主需提供下面信息 1.用的哪个版本 2.去看下日志 3.具体如何配置的 |
|
本帖最后由 Hentai 于 2016-12-26 16:53 编辑 就算把memory调小都没有用！一样过了一会就自动停止了 |
| 不管我是不是用excu source都是这样 运行一会就停了 |
/2 
Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.
简书 /
京ICP备2020039040号
简书网举报电话：021-34700000
组图打开中，请稍候......