分享

启动spark-shell一直产生ERROR TaskSchedulerImpl: Lost an executor 27 (already ...

唐运 发表于 2014-12-18 14:52:10 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 5 57662
spark-shell进入后,一直出现下列日志,不知道怎么回事,请大牛给讲解下。

scala> 14/12/18 14:48:39 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:58006/user/Executor#-778656424] with ID 25
14/12/18 14:48:40 INFO SparkDeploySchedulerBackend: Executor 25 disconnected, so removing it
14/12/18 14:48:40 ERROR TaskSchedulerImpl: Lost an executor 25 (already removed): remote Akka client disassociated
14/12/18 14:48:40 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/25 is now EXITED (Command exited with code 53)
14/12/18 14:48:40 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/25 removed: Command exited with code 53
14/12/18 14:48:40 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/26 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/26 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:40 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/26 is now LOADING
14/12/18 14:48:40 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/26 is now RUNNING
14/12/18 14:48:43 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:44592/user/Executor#1296942457] with ID 26
14/12/18 14:48:43 INFO SparkDeploySchedulerBackend: Executor 26 disconnected, so removing it
14/12/18 14:48:43 ERROR TaskSchedulerImpl: Lost an executor 26 (already removed): remote Akka client disassociated
14/12/18 14:48:43 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/26 is now EXITED (Command exited with code 53)
14/12/18 14:48:43 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/26 removed: Command exited with code 53
14/12/18 14:48:43 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/27 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:43 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/27 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:43 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/27 is now LOADING
14/12/18 14:48:44 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/27 is now RUNNING
14/12/18 14:48:46 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:46979/user/Executor#-204545127] with ID 27
14/12/18 14:48:47 INFO SparkDeploySchedulerBackend: Executor 27 disconnected, so removing it
14/12/18 14:48:47 ERROR TaskSchedulerImpl: Lost an executor 27 (already removed): remote Akka client disassociated
14/12/18 14:48:47 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/27 is now EXITED (Command exited with code 53)
14/12/18 14:48:47 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/27 removed: Command exited with code 53
14/12/18 14:48:47 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/28 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:47 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/28 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:47 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/28 is now LOADING
14/12/18 14:48:47 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/28 is now RUNNING
14/12/18 14:48:50 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:40004/user/Executor#-1976174657] with ID 28
14/12/18 14:48:50 INFO SparkDeploySchedulerBackend: Executor 28 disconnected, so removing it
14/12/18 14:48:50 ERROR TaskSchedulerImpl: Lost an executor 28 (already removed): remote Akka client disassociated
14/12/18 14:48:50 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/28 is now EXITED (Command exited with code 53)
14/12/18 14:48:50 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/28 removed: Command exited with code 53
14/12/18 14:48:50 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/29 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:50 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/29 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:50 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/29 is now LOADING
14/12/18 14:48:51 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/29 is now RUNNING
14/12/18 14:48:53 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:48870/user/Executor#1714635940] with ID 29
14/12/18 14:48:54 INFO SparkDeploySchedulerBackend: Executor 29 disconnected, so removing it
14/12/18 14:48:54 ERROR TaskSchedulerImpl: Lost an executor 29 (already removed): remote Akka client disassociated
14/12/18 14:48:54 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/29 is now EXITED (Command exited with code 53)
14/12/18 14:48:54 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/29 removed: Command exited with code 53
14/12/18 14:48:54 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/30 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:54 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/30 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:54 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/30 is now LOADING
14/12/18 14:48:55 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/30 is now RUNNING
14/12/18 14:48:57 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:53581/user/Executor#-296954132] with ID 30
14/12/18 14:48:58 INFO SparkDeploySchedulerBackend: Executor 30 disconnected, so removing it
14/12/18 14:48:58 ERROR TaskSchedulerImpl: Lost an executor 30 (already removed): remote Akka client disassociated
14/12/18 14:48:58 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/30 is now EXITED (Command exited with code 53)
14/12/18 14:48:58 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/30 removed: Command exited with code 53
14/12/18 14:48:58 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/31 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:48:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/31 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:48:58 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/31 is now LOADING
14/12/18 14:48:58 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/31 is now RUNNING
14/12/18 14:49:01 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:43484/user/Executor#-2084341174] with ID 31
14/12/18 14:49:01 INFO SparkDeploySchedulerBackend: Executor 31 disconnected, so removing it
14/12/18 14:49:01 ERROR TaskSchedulerImpl: Lost an executor 31 (already removed): remote Akka client disassociated
14/12/18 14:49:01 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/31 is now EXITED (Command exited with code 53)
14/12/18 14:49:01 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/31 removed: Command exited with code 53
14/12/18 14:49:01 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/32 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:49:01 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/32 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:49:01 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/32 is now LOADING
14/12/18 14:49:02 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/32 is now RUNNING
14/12/18 14:49:04 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@dn3.9961.bj:48902/user/Executor#768818777] with ID 32
14/12/18 14:49:05 INFO SparkDeploySchedulerBackend: Executor 32 disconnected, so removing it
14/12/18 14:49:05 ERROR TaskSchedulerImpl: Lost an executor 32 (already removed): remote Akka client disassociated
14/12/18 14:49:05 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/32 is now EXITED (Command exited with code 53)
14/12/18 14:49:05 INFO SparkDeploySchedulerBackend: Executor app-20141218144713-0056/32 removed: Command exited with code 53
14/12/18 14:49:05 INFO AppClient$ClientActor: Executor added: app-20141218144713-0056/33 on worker-20141203180353-dn3.9961.bj-45017 (dn3.9961.bj:45017) with 8 cores
14/12/18 14:49:05 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141218144713-0056/33 on hostPort dn3.9961.bj:45017 with 8 cores, 512.0 MB RAM
14/12/18 14:49:05 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/33 is now LOADING
14/12/18 14:49:05 INFO AppClient$ClientActor: Executor updated: app-20141218144713-0056/33 is now RUNNING

已有(5)人评论

跳转到指定楼层
bioger_hit 发表于 2014-12-22 11:47:41
楼主列出你的具体问题
回复

使用道具 举报

muyannian 发表于 2014-12-22 11:58:46
spark启动方式是什么
回复

使用道具 举报

muyannian 发表于 2014-12-22 12:08:03
确保下面是正常的:
1.7077是否可以访问

2.配置下面变量
spark-env.sh

export MASTER=spark://<master-ip>:7077

3.测试下单节点是否可以启动


回复

使用道具 举报

唐运 发表于 2014-12-24 17:34:24
启动都没问题,是因为一个节点的磁盘慢了导致的。谢谢大家关心
回复

使用道具 举报

liuzhixin137 发表于 2016-6-1 14:42:17
唐运 发表于 2014-12-24 17:34
启动都没问题,是因为一个节点的磁盘慢了导致的。谢谢大家关心

你好,我是出现一样的情况,总是lose executor   。至于您说的一个节点的磁盘慢了导致,请问是如何解决的呢?
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条