请教一下为什么flume每次都是运行一会就停了
本帖最后由 pig2 于 2016-12-26 19:19 编辑请各位大神赐教
不管我是不是用excu source都是这样 运行一会就停了 本帖最后由 Hentai 于 2016-12-26 16:53 编辑
就算把memory调小都没有用！一样过了一会就自动停止了 Hentai 发表于 2016-12-26 16:46
就算把memory调小都没有用！一样过了一会就自动停止了
没遇到过，楼主需提供下面信息
1.用的哪个版本
2.去看下日志
3.具体如何配置的
langke93 发表于 2016-12-26 19:24
没遇到过，楼主需提供下面信息
1.用的哪个版本
2.去看下日志
配置
hdfs.sinks.fs.channel=fc
hdfs.sources.fr.channels=fc
hdfs.sources.fr.type=exec
hdfs.sources.fr.command=tshark -i em2 -t ad
hdfs.sinks.fs.type=hdfs
hdfs.sinks.fs.hdfs.path=hdfs://10.0.40.2:8020/user/xiakang
#hdfs.sinks.fs.hdfs.filePrefix=Syslog
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
hdfs.sinks.fs.hdfs.rollCount=0
hdfs.sinks.fs.hdfs.fileType=DataStream
hdfs.sinks.fs.hdfs.minBlockReplicas=1
hdfs.sinks.fs.hdfs.inUseSuffix=log
hdfs.channels.fc.type=SPILLABLEMEMORY
hdfs.channels.fc.checkpointDir = /mnt/flume/checkpoint
hdfs.channels.fc.dataDirs = /mnt/flume/data
版本 Flume 1.5.2.2.5.0.0-1245
错误
Failed while running command: tshark -i em2 -t ad
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:331)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
26 Dec 2016 16:23:41,039 INFO (org.apache.flume.source.ExecSource$ExecRu
nnable.run:370)- Command exited with 141
26 Dec 2016 16:23:41,040 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:149)- Component type: SOURCE, name: fr stopped
26 Dec 2016 16:23:41,040 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:155)- Shutdown Metric for type: SOURCE, name: fr. source.start.tim
e == 1482740567800
26 Dec 2016 16:23:41,040 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:161)- Shutdown Metric for type: SOURCE, name: fr. source.stop.time
== 1482740621040
26 Dec 2016 16:23:41,041 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.append-batch
.accepted == 0
26 Dec 2016 16:23:41,041 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.append-batch
.received == 0
26 Dec 2016 16:23:41,041 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.append.accep
ted == 0
26 Dec 2016 16:23:41,041 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.append.recei
ved == 0
26 Dec 2016 16:23:41,041 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.events.accep
ted == 514320
26 Dec 2016 16:23:41,042 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.events.recei
ved == 514334
26 Dec 2016 16:23:41,042 INFO (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)- Shutdown Metric for type: SOURCE, name: fr. src.open-connect
ion.count == 0
26 Dec 2016 16:23:41,042 ERROR (org.apache.
flume.sink.hdfs.HDFSEventSink.process:466)- process failed
java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout
might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket
Writer.java:687)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
26 Dec 2016 16:23:41,043 ERROR (org.apache.
flume.SinkRunner$PollingRunner.run:160)- Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before H
DFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too
long.
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:470)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.c
allTimeout might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket
Writer.java:687)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448)
... 3 more
Hentai 发表于 2016-12-27 09:24
配置
hdfs.sinks.fs.channel=fc
hdfs.sources.fr.channels=fc
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
为何这两个设置这么大。超时出现问题正常的设置小一点，根据情况自己调试下。
建议初始值：
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次：
20
40
80
200
等值。
页:
[1]