分享

spark中提交任务总是不停的循环(如内容)

不可替代 发表于 2018-4-2 15:51:05 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 4 13823
本帖最后由 不可替代 于 2018-4-2 15:53 编辑

18/04/02 15:48:31 INFO storage.BlockManagerMaster: Removal of executor 29 requested
18/04/02 15:48:31 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 29
18/04/02 15:48:31 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 29 from BlockManagerMaster.
18/04/02 15:48:33 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
18/04/02 15:48:37 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 30 for reason Container marked as failed: container_e49_1522654847559_0001_02_000034 on host: Iwanvi19. Exit status: -1000. Diagnostics: Not able to initialize app-cache directories in any of the configured local directories for user root

18/04/02 15:48:37 INFO storage.BlockManagerMaster: Removal of executor 30 requested
18/04/02 15:48:37 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 30
18/04/02 15:48:37 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 30 from BlockManagerMaster.
18/04/02 15:48:43 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 31 for reason Container marked as failed: container_e49_1522654847559_0001_02_000035 on host: Iwanvi19. Exit status: -1000. Diagnostics: Not able to initialize app-cache directories in any of the configured local directories for user root

希望大神指点指定
提交语句:./spark-submit  --class com.book.bigcarbon.BigCarbonStatic  --jars ~/cartest.jar 10 --driver-memory 1g --executor-cores 4 --executor-memory 1g      

已有(4)人评论

跳转到指定楼层
hello2018 发表于 2018-4-2 18:54:05
可能内存配置过小造成的。配置spark-env.sh的内存
参考这个帖子
spark程序遇到了这样的问题 Initial job has not accepted any resources
http://www.aboutyun.com/forum.php?mod=viewthread&tid=14907



回复

使用道具 举报

不可替代 发表于 2018-4-2 23:39:52
hello2018 发表于 2018-4-2 18:54
可能内存配置过小造成的。配置spark-env.sh的内存
参考这个帖子
spark程序遇到了这样的问题 Initial job  ...

谢谢!不过我调整了也是一样的,我的三台21G测试机


回复

使用道具 举报

lyuan 发表于 2018-5-4 16:40:44
你的日志提示这样的错误:“18/04/02 15:48:33 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources”。提示你资源不够,你打开spark的web页面,再跑一次你这个脚本,看看资源消耗情况,如果脚本没有跑完,资源就耗光了,你再修改资源参数
回复

使用道具 举报

lyuan 发表于 2018-5-4 16:42:19
你的日志提示这样的错误:“18/04/02 15:48:33 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources”。提示你资源不够,你打开spark的web页面,再跑一次你这个脚本,看看资源消耗情况,如果脚本没有跑完,资源就耗光了,你再修改资源参数
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条