这篇文章是有问题的吧！源码来看，是不能同时做一台机器的负载跟容错的，因为你他group加载配置的时候，不会加载重复的sink，就比如上面那个兄弟的报错。

#配置文件：load_sink_case14.conf # Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2 k3 k4
a1.channels = c1 c2 c3 c4

a1.sinkgroups = g1 g2 g3

a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type =load_balance
a1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector =round_robin  

a1.sinkgroups.g2.sinks = k1 k3
a1.sinkgroups.g2.processor.type= failover
a1.sinkgroups.g2.processor.priority.k1= 5
a1.sinkgroups.g2.processor.priority.k3= 10
a1.sinkgroups.g2.processor.maxpenalty= 10000

a1.sinkgroups.g3.sinks = k2 k4
a1.sinkgroups.g3.processor.type= failover
a1.sinkgroups.g3.processor.priority.k2= 5
a1.sinkgroups.g3.processor.priority.k4= 10
a1.sinkgroups.g3.processor.maxpenalty= 10000

# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 50000
a1.sources.r1.host = 192.168.4.23
a1.sources.r1.channels = c1 c2 c3 c4
  
# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 192.168.4.21
a1.sinks.k1.port = 50000
   
a1.sinks.k2.type = avro
a1.sinks.k2.channel = c2
a1.sinks.k2.hostname = 192.168.4.22
a1.sinks.k2.port = 50000

a1.sinks.k3.type = avro
a1.sinks.k3.channel = c3
a1.sinks.k3.hostname = 192.168.4.25
a1.sinks.k3.port = 50000

a1.sinks.k4.type = avro
a1.sinks.k4.channel = c4
a1.sinks.k4.hostname = 192.168.4.26
a1.sinks.k4.port = 50000


# Use a channel which buffers events inmemory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

a1.channels.c3.type = memory
a1.channels.c3.capacity = 1000
a1.channels.c3.transactionCapacity = 100

a1.channels.c4.type = memory
a1.channels.c4.capacity = 1000
a1.channels.c4.transactionCapacity = 100
配置负载均衡和容错，报这个错误，求指导，哪里配置有问题啊？

694809295@qq.co 发表于 2015-3-19 18:28
楼主，我试着把rollsize设置成1M或者10M分割一次，速度上也没什么提升，有时候控制台只是打出一些debug日志 ...

修改一下flume-env.sh，修改JAVA_OPTS="-Xms100m -Xmx1024m -Dcom.sun.management.jmxremote"

694809295@qq.co 发表于 2015-3-19 18:28
楼主，我试着把rollsize设置成1M或者10M分割一次，速度上也没什么提升，有时候控制台只是打出一些debug日志 ...

是不是使用的memory的方式。
内存不够了。
更换其他方式试试

楼主，我试着把rollsize设置成1M或者10M分割一次，速度上也没什么提升，有时候控制台只是打出一些debug日志，hdfs没有新的数据写入。请问你有没有类似 a 是source 到b是collector c是hdfs的配置文件，参考下，就像我上面的那种结构，反复测试各种参数 性能上就是提不上去，flume自带的conf；里的内存设置到了2-10G log4j里面的级别也是error的了 。主要会报java.lang.OutOfMemoryError: GC overhead limit exceeded；
Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue ；
nable to deliver event. Exception follows.org.apache.flume.EventDeliveryException: Failed to send events processing.；
主要会报这些错误，求解。万分感谢

我自己搭建了一下，有点问题想请教，谢谢。详情请移步我的日志http://www.aboutyun.com/blog-13274-1798.html

请问楼主一个flume采集数据到hdfs性能问题  将n10作为数据源 n08作为collector n07是hdfs n10的配置大概

n10的flume配置文件如下
# Name the components on this agent
a1.sources = r1  
a1.sinks = k1  
a1.channels = c1

# Describe the sink
a1.sinks.k1.type = logger  

####
a1.sources.r1.type = spooldir 
a1.sources.r1.spoolDir = /home/nids/wg/apache-flume-1.5.2-bin/ceshi12
a1.sources.r1.fileHeader =false
a1.sources.r1.channels = c1
####

# Describe/configure the source
#a1.sources.r1.type = avro   
a1.sources.r1.bind = localhost  
a1.sources.r1.port = 44444 

# avro sink   
a1.sinks.k1.type = avro  
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = r09n08  
a1.sinks.k1.port = 55555
 
# Use a channel which buffers events in file
a1.channels = c1
a1.channels.c1.type = memory
#a1.channels.c1.checkpointDir = /home/nids/wg/apache-flume-1.5.2-bin/checkpoint
#a1.channels.c1.dataDirs = /home/nids/wg/apache-flume-1.5.2-bin/datadir
 
a1.sinks.k1.hdfs.batchSize = 10000 
#a1.channels.c1.type = memory  
a1.channels.c1.capacity = 100000  
a1.channels.c1.transactionCapacity = 10000  

# Bind the source and sink to the channel
a1.sources.r1.channels = c1  
a1.sinks.k1.channel = c1

==================
n08的flume配置文件如下
# Name the components on this agent
a1.sources = r1  
a1.sinks = k1  
a1.channels = c1  

# Describe/configure the source
a1.sources.r1.type = avro  
a1.sources.r1.bind = r09n08  
a1.sources.r1.port = 55555   
a1.sources.r1.interceptors = i1  
a1.sources.r1.interceptors.i1.type = timestamp   

#hdfs sink
a1.sinks.k1.type = hdfs  
a1.sinks.k1.hdfs.path = hdfs://r09n07:8020/project/dame/input/%Y%m%d/%H
a1.sinks.k1.hdfs.fileType = DataStream 
a1.sinks.k1.hdfs.filePrefix = hdfs-
a1.sinks.k1.hdfs.rollInterval = 0 
#a1.sinks.k1.hdfs.fileSuffix = .log  
#a1.sinks.k1.hdfs.round = true  
#a1.sinks.k1.hdfs.roundValue = 1  
#a1.sinks.k1.hdfs.roundUnit = minute  
a1.sinks.k1.hdfs.rollSize = 67108864  
a1.sinks.k1.hdfs.rollCount = 0    
#a1.sinks.k1.hdfs.writeFormat = Text  
 
# Use a channel which buffers events in file
a1.channels = c1
a1.channels.c1.type = memory
#a1.channels.c1.checkpointDir=/home/nids/wg/apache-flume-1.5.2-bin/checkpoint
#a1.channels.c1.dataDirs=/home/nids/wg/apache-flume-1.5.2-bin/datadir
 
 
a1.sinks.k1.hdfs.batchSize = 10000
#a1.sinks.k1.hdfs.callTimeout = 6000
#a1.sinks.k1.hdfs.appendTimeout = 6000   
#a1.channels.c1.type = memory  
a1.channels.c1.capacity = 100000  
a1.channels.c1.transactionCapacity = 10000 
a1.sources.r1.channels = c1  
a1.sinks.k1.channel = c1
===================
复制代码

采集的数据文件才几十m都写了好久，有时只写了一部分就不动了。channel是memory类型，在channel的几个参数都来回设置了好多次也没什么起色transactionCapacity  capacity 还有sink的batchSize。我是用文件大小类分割的大概64M  a1.sinks.k1.hdfs.rollSize = 67108864   请问具体是什么地方设置出的问题，谢谢

好文章，学习了