分享

spark-submit运行程序报错

xw2016 发表于 2016-6-2 22:25:35 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 3 9625
运行spark-shell没问题,运行spark-submit出错,操作如下:
进到spark目录:bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://yun01-nn-01:7077,yun01-nn-02:7077  lib/spark-examples-1.5.2-hadoop2.6.0.jar 100
日志:
16/06/01 23:53:49 INFO Worker: Registered signal handlers for [TERM, HUP, INT]
16/06/01 23:53:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/01 23:53:50 INFO SecurityManager: Changing view acls to: hadoop
16/06/01 23:53:50 INFO SecurityManager: Changing modify acls to: hadoop
16/06/01 23:53:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
16/06/01 23:53:52 INFO Slf4jLogger: Slf4jLogger started
16/06/01 23:53:52 INFO Remoting: Starting remoting
16/06/01 23:53:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@192.168.56.14:51528]
16/06/01 23:53:52 INFO Utils: Successfully started service 'sparkWorker' on port 51528.
16/06/01 23:53:52 INFO Worker: Starting Spark worker 192.168.56.14:51528 with 1 cores, 1871.0 MB RAM
16/06/01 23:53:52 INFO Worker: Running Spark version 1.5.2
16/06/01 23:53:52 INFO Worker: Spark home: /application/hadoop/spark
16/06/01 23:53:53 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
16/06/01 23:53:53 INFO WorkerWebUI: Started WorkerWebUI at http://192.168.56.14:8081
16/06/01 23:53:53 INFO Worker: Connecting to master yun01-nn-01:7077...
16/06/01 23:53:53 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@yun01-nn-01:7077] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://sparkMaster@yun01-nn-01:7077]] Caused by: [Connection refused: yun01-nn-01/192.168.56.11:7077]
16/06/01 23:53:53 WARN Worker: Failed to connect to master yun01-nn-01:7077
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster@yun01-nn-01:7077/), Path(/user/Master)]
        at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
        at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
        at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
        at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:120)
        at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
        at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:266)
        at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:533)
        at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:569)
        at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:559)
        at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
        at akka.remote.EndpointWriter.postStop(Endpoint.scala:557)
        at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
        at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:411)
        at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
        at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
        at akka.actor.ActorCell.terminate(ActorCell.scala:369)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

已有(3)人评论

跳转到指定楼层
xw2016 发表于 2016-6-2 22:26:48
网上找到了一些同样报错的,但很少有人回答,求助各位大神。
回复

使用道具 举报

einhep 发表于 2016-6-3 14:33:25
xw2016 发表于 2016-6-2 22:26
网上找到了一些同样报错的,但很少有人回答,求助各位大神。

7077端口是不是被暂用了。这是端口连接不上
回复

使用道具 举报

xw2016 发表于 2016-6-3 16:34:39
einhep 发表于 2016-6-3 14:33
7077端口是不是被暂用了。这是端口连接不上

应该不是,我前面运行spark-shell还可以的。后面用spark-submit就不行了
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条