本帖最后由 ansonmusicer 于 2016-4-15 18:15 编辑
最近把spark从1.5升级到1.6,发现原来的streaming代码不能用了,代码如下:
[mw_shl_code=java,true]JavaReceiverInputDStream<SparkFlumeEvent> stream1 = FlumeUtils.createStream(jssc, "0.0.0.0", 44444, StorageLevel.MEMORY_ONLY());
JavaReceiverInputDStream<SparkFlumeEvent> stream2 = FlumeUtils.createStream(jssc, "0.0.0.0", 44444, StorageLevel.MEMORY_ONLY());
JavaReceiverInputDStream<SparkFlumeEvent> stream3 = FlumeUtils.createStream(jssc, "0.0.0.0", 55555, StorageLevel.MEMORY_ONLY());
JavaReceiverInputDStream<SparkFlumeEvent> stream4 = FlumeUtils.createStream(jssc, "0.0.0.0", 55555, StorageLevel.MEMORY_ONLY());[/mw_shl_code]
提交语句如下:
spark-submit --master yarn-client --num-executors 2 --executor-cores 4 --driver-cores 4
在1.5版本下,上面的代码运行正常,两个executor,每个executor监听了两个flume端口
但升级到1.6后,发现只有一个端口被监听了,只有写成 --num-executors 4 这样才能正常运行,也就是需要4个executor,每个executor只能监听一个端口,我有的端口平时流量很少,如果每个端口都要配一个executor,那岂不是很浪费?
请问有没有什么方法解决这个问题,在1.6里面让多个端口监听共享一个executor?
谢谢!
|