启动spark报错，求解决

丹青穆怀 · 发表于 2017-3-7 15:10:02

nextuser 发表于 2017-3-7 14:28
上图吧，你运行的用户是hadoop，还是root用户。
不行尝试下本地路径
spark.eventLog.dir ...

./spark-shell
Welcome to
   ____             __
   / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.2
   /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
17/03/07 15:06:44 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
17/03/07 15:07:26 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@655d7d21 rejected from java.util.concurrent.ThreadPoolExecutor@5cc23584[Running, pool size = 2, active threads = 1, queued tasks = 0, completed tasks = 3]
      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
      at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132)
      at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
      at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

出现了新的错误。。。。

lenovoyufeilil · 发表于 2017-3-7 15:12:12

丹青穆怀发表于 2017-3-7 14:37
不需要吧，已经配置了HA了

需要啊,你hdfs的路径写得不对当然找不到了

einhep · 发表于 2017-3-7 17:49:36

lenovoyufeilil 发表于 2017-3-7 15:12
需要啊,你hdfs的路径写得不对当然找不到了

spark哪个版本？尝试

将SPARK_MASTER_IP=master改成

SPARK_MASTER_IP=10.9.2.100

w517424787 · 发表于 2017-3-8 09:01:24

log提示不能用本地路径了，你还是将刚才的那个路径改成HDFS的上的路径：hdfs://namenode:port/beh/SparkeventLog ,如果没有端口，就直接写成 hdfs://namenode/beh/SparkeventLog就好了！

丹青穆怀 · 发表于 2017-3-8 17:41:37

w517424787 发表于 2017-3-8 09:01
log提示不能用本地路径了，你还是将刚才的那个路径改成HDFS的上的路径：hdfs://namenode:port/beh/Sparkeve ...

改成这样也不行，报错
ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From hadoop001/172.16.13.224 to hadoop001:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused