分享

spark on yarn 运行问题,一直循环

xw2016 发表于 2016-7-9 22:35:23 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 1 13154
spark on yarn 运行命令后,一直循环出现:Application report for application_1468011021285_0005 (state: RUNNING),如下面日志所示:

[hadoop@yun01-dn-01 spark]$ bin/spark-submit --master yarn-cluster --executor-memory 1g --class sparkscala.k3.app.Query /application/hadoop/testjar/query.jar  hdfs://ns1/data/log1.txt hdfs://ns1/sparkout/query4
16/07/09 06:11:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/09 06:11:14 INFO Client: Requesting a new application from cluster with 3 NodeManagers
16/07/09 06:11:14 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/07/09 06:11:14 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/07/09 06:11:14 INFO Client: Setting up container launch context for our AM
16/07/09 06:11:14 INFO Client: Setting up the launch environment for our AM container
16/07/09 06:11:14 WARN Client: SPARK_JAR detected in the system environment. This variable has been deprecated in favor of the spark.yarn.jar configuration variable.
16/07/09 06:11:14 INFO Client: Preparing resources for our AM container
16/07/09 06:11:15 WARN Client: SPARK_JAR detected in the system environment. This variable has been deprecated in favor of the spark.yarn.jar configuration variable.
16/07/09 06:11:15 INFO Client: Uploading resource file:/application/hadoop/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1468011021285_0005/spark-assembly-1.5.2-hadoop2.6.0.jar
16/07/09 06:11:26 INFO Client: Uploading resource file:/application/hadoop/testjar/query.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1468011021285_0005/query.jar
16/07/09 06:11:27 INFO Client: Uploading resource file:/tmp/spark-47438339-ed4f-41f5-be4e-dfe5cce71101/__spark_conf__5752978701195266719.zip -> hdfs://ns1/user/hadoop/.sparkStaging/application_1468011021285_0005/__spark_conf__5752978701195266719.zip
16/07/09 06:11:27 INFO SecurityManager: Changing view acls to: hadoop
16/07/09 06:11:27 INFO SecurityManager: Changing modify acls to: hadoop
16/07/09 06:11:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
16/07/09 06:11:28 INFO Client: Submitting application 5 to ResourceManager
16/07/09 06:11:28 INFO YarnClientImpl: Submitted application application_1468011021285_0005
16/07/09 06:11:29 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:29 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1468015895663
         final status: UNDEFINED
         tracking URL: http://yun01-nn-01:8088/proxy/application_1468011021285_0005/
         user: hadoop
16/07/09 06:11:30 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:31 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:32 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:33 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:34 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:35 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:36 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:37 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:38 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:39 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:40 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:41 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:42 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:43 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:44 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:45 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:46 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:47 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:48 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:49 INFO Client: Application report for application_1468011021285_0005 (state: ACCEPTED)
16/07/09 06:11:50 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:50 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.56.14
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1468015895663
         final status: UNDEFINED
         tracking URL: http://yun01-nn-01:8088/proxy/application_1468011021285_0005/
         user: hadoop
16/07/09 06:11:51 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:52 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:53 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:54 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:55 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:56 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:57 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:58 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:11:59 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:12:00 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:12:01 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:12:03 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)
16/07/09 06:12:04 INFO Client: Application report for application_1468011021285_0005 (state: RUNNING)




已有(1)人评论

跳转到指定楼层
xw2016 发表于 2016-7-16 17:32:30
问题已解决,原来没有问题,只是运行时间有点久。
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条