spark-shell 连接集群出错。求大神指点

./spark-shell --master spark://Master:7077 --executor-memory 1024m --driver-memory 1024m

执行结果出错：
17/03/02 12:37:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/03/02 12:37:40 INFO spark.SecurityManager: Changing view acls to: hadoop
17/03/02 12:37:40 INFO spark.SecurityManager: Changing modify acls to: hadoop
17/03/02 12:37:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/03/02 12:37:41 INFO spark.HttpServer: Starting HTTP Server
17/03/02 12:37:41 INFO server.Server: jetty-8.y.z-SNAPSHOT
17/03/02 12:37:41 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:40136
17/03/02 12:37:41 INFO util.Utils: Successfully started service 'HTTP class server' on port 40136.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (OpenJDK Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
17/03/02 12:37:47 INFO spark.SparkContext: Running Spark version 1.6.0
17/03/02 12:37:48 INFO spark.SecurityManager: Changing view acls to: hadoop
17/03/02 12:37:48 INFO spark.SecurityManager: Changing modify acls to: hadoop
17/03/02 12:37:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/03/02 12:37:48 INFO util.Utils: Successfully started service 'sparkDriver' on port 38286.
17/03/02 12:37:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/03/02 12:37:49 INFO Remoting: Starting remoting
17/03/02 12:37:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@127.0.0.1:47787]
17/03/02 12:37:49 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 47787.
17/03/02 12:37:49 INFO spark.SparkEnv: Registering MapOutputTracker
17/03/02 12:37:49 INFO spark.SparkEnv: Registering BlockManagerMaster
17/03/02 12:37:49 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-d6835a89-4165-4003-b7b8-ef21c655153e
17/03/02 12:37:49 INFO storage.MemoryStore: MemoryStore started with capacity 517.4 MB
17/03/02 12:37:49 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/03/02 12:37:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
17/03/02 12:37:50 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
17/03/02 12:37:50 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/03/02 12:37:50 INFO ui.SparkUI: Started SparkUI at http://127.0.0.1:4040
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Connecting to master spark://Master:7077...
17/03/02 12:37:50 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20170302123750-0004
17/03/02 12:37:50 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37168.
17/03/02 12:37:50 INFO netty.NettyBlockTransferService: Server created on 37168
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor added: app-20170302123750-0004/0 on worker-20170302103714-127.0.0.1-34196 (127.0.0.1:34196) with 1 cores
17/03/02 12:37:50 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/03/02 12:37:50 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170302123750-0004/0 on hostPort 127.0.0.1:34196 with 1 cores, 1024.0 MB RAM
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor added: app-20170302123750-0004/1 on worker-20170302103714-127.0.0.1-53686 (127.0.0.1:53686) with 1 cores
17/03/02 12:37:50 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170302123750-0004/1 on hostPort 127.0.0.1:53686 with 1 cores, 1024.0 MB RAM
17/03/02 12:37:50 INFO storage.BlockManagerMasterEndpoint: Registering block manager 127.0.0.1:37168 with 517.4 MB RAM, BlockManagerId(driver, 127.0.0.1, 37168)
17/03/02 12:37:50 INFO storage.BlockManagerMaster: Registered BlockManager
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor added: app-20170302123750-0004/2 on worker-20170302103715-127.0.0.1-32909 (127.0.0.1:32909) with 1 cores
17/03/02 12:37:50 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170302123750-0004/2 on hostPort 127.0.0.1:32909 with 1 cores, 1024.0 MB RAM
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/2 is now RUNNING
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/0 is now RUNNING
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/1 is now RUNNING
17/03/02 12:37:51 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/03/02 12:37:51 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
17/03/02 12:37:53 INFO repl.SparkILoop: Created sql context..
SQL context available as sqlContext.
17/03/02 12:37:54 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/1 is now EXITED (Command exited with code 1)
17/03/02 12:37:54 INFO cluster.SparkDeploySchedulerBackend: Executor app-20170302123750-0004/1 removed: Command exited with code 1

scala> 17/03/02 12:39:54 WARN netty.NettyRpcEndpointRef: Error sending message [message = RemoveExecutor(1,Command exited with code 1)] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:359)
at org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.executorRemoved(SparkDeploySchedulerBackend.scala:144)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$receive$1.applyOrElse(AppClient.scala:186)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 12 more

qcbb001 发表于 2017-3-12 10:04
SPARK_MASTER_IP如何配置的

spark.env.sh:
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
export SPARK_LOCAL_IP="127.0.0.1"
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.i386
export SPARK_MASTER_IP=192.168.137.128         //master地址
export SCALA_HOME=/usr/lib/scala/scala-2.11.8
export SPARK_WORKER_MEMORY=2g
export HADOOP_CONF_DIR=/usr/local/hadoop/ect/hadoop


hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.128 Master
192.168.137.130 Slave1
192.168.137.129 Slave2

SPARK_MASTER_IP如何配置的

海滩发红 发表于 2017-3-12 12:05
spark.env.sh:
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
export SPARK ...

netty是spark通信框架，通信超时了。所以产生问题。楼主把：：1也就是ip6先注释掉。

hyj 发表于 2017-3-12 12:46
netty是spark通信框架，通信超时了。所以产生问题。楼主把：：1也就是ip6先注释掉。

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.128 Master
192.168.137.130 Slave1
192.168.137.129 Slave2

这样子吗？

运行之后，还是会有这个问题。

超时分为多种，ip6是一个可能原因，超时时间，楼主也可以设置下
SparkConf: conf.set("spark.rpc.askTimeout", "600s")
  spark-defaults.conf: spark.rpc.askTimeout 600s
spark-submit: --conf spark.rpc.askTimeout=600s

qcbb001 发表于 2017-3-12 13:37
超时分为多种，ip6是一个可能原因，超时时间，楼主也可以设置下
SparkConf: conf.set("spark.rpc.askTime ...

不知道大神有没有看到这个：
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/2 is now RUNNING
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/0 is now RUNNING
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/1 is now RUNNING
17/03/02 12:37:51 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/03/02 12:37:51 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
17/03/02 12:37:53 INFO repl.SparkILoop: Created sql context..
SQL context available as sqlContext.
17/03/02 12:37:54 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170302123750-0004/1 is now EXITED (Command exited with code 1)
17/03/02 12:37:54 INFO cluster.SparkDeploySchedulerBackend: Executor app-20170302123750-0004/1 removed: Command exited with code 1




想问一下，这个问题是怎么产生的

qcbb001 发表于 2017-3-12 13:37
超时分为多种，ip6是一个可能原因，超时时间，楼主也可以设置下
SparkConf: conf.set("spark.rpc.askTime ...

大神您好……我为小白……你说的那些要怎么做……。具体步骤。
谢谢

海滩发红 发表于 2017-3-12 13:40
不知道大神有没有看到这个：
17/03/02 12:37:50 INFO client.AppClient$ClientEndpoint: Ex ...

这是正常信息，应该不是错误

einhep 发表于 2017-3-12 19:34
这是正常信息，应该不是错误

可是他会一直重复这个：
17/03/02 12:39:54 WARN netty.NettyRpcEndpointRef: Error sending message [message = RemoveExecutor(1,Command exited with code 1)] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:359)
at org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.executorRemoved(SparkDeploySchedulerBackend.scala:144)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$receive$1.applyOrElse(AppClient.scala:186)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 12 more