分享

spark Phoenix

zstu 发表于 2016-10-24 20:27:49 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 4 16213
我用Phoenix存储spark计算的结果,在本地local模式下运行可以,但我用spark-submit提交到集群运行后,spark-submit --class com.bigdata.main.Main --master yarn --driver-memory 2G --executor-memory 2G --num-executors 10 /home/test/Test.jar ,报错:Caused by: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found

Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)    at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232)    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:971)    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:903)    at org.apache.phoenix.spark.ProductRDDFunctions.saveToPhoenix(ProductRDDFunctions.scala:51)    at com.mypackage.save(DAOImpl.scala:41)    at com.mypackage.ProtoStreamingJob.execute(ProtoStreamingJob.scala:58)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:606)    at com.mypackage.SparkApplication.sparkRun(SparkApplication.scala:95)    at com.mypackage.SparkApplication$delayedInit$body.apply(SparkApplication.scala:112)    at scala.Function0$class.apply$mcV$sp(Function0.scala:40)    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)    at scala.App$$anonfun$main$1.apply(App.scala:71)    at scala.App$$anonfun$main$1.apply(App.scala:71)    at scala.collection.immutable.List.foreach(List.scala:318)    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)    at scala.App$class.main(App.scala:71)    at com.mypackage.SparkApplication.main(SparkApplication.scala:15)    at com.mypackage.ProtoStreamingJobRunner.main(ProtoStreamingJob.scala)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:606)    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)    ... 30 more


后来我在/etc/spark/conf下的classpath.txt中添加phoenix-core**.jar后,又报错:
java.lang.NoClassDefFoundError: co/cask/tephra/TransactionSystemClient
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:221)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:135)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:98)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:82)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:70)
        at org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getUpsertColumnMetadataList(PhoenixConfigurationUtil.java:230)
        at org.apache.phoenix.spark.ProductRDDFunctions$$anonfun$1.apply(ProductRDDFunctions.scala:41)
        at org.apache.phoenix.spark.ProductRDDFunctions$$anonfun$1.apply(ProductRDDFunctions.scala:37)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: co.cask.tephra.TransactionSystemClient
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
请问大神这个这么解决?

已有(4)人评论

跳转到指定楼层
einhep 发表于 2016-10-24 21:43:45
应该还是缺包添加这个试试

phoenix-**-client-**-hbase.jar  

回复

使用道具 举报

zstu 发表于 2016-10-24 21:44:48
einhep 发表于 2016-10-24 21:43
应该还是缺包添加这个试试

phoenix-**-client-**-hbase.jar

在spark-submit 的-jars中添加吗
回复

使用道具 举报

zstu 发表于 2016-10-25 10:42:41
zstu 发表于 2016-10-24 21:44
在spark-submit 的-jars中添加吗

加上去,还是报同样的错误
回复

使用道具 举报

langke93 发表于 2016-10-25 19:06:06
zstu 发表于 2016-10-25 10:42
加上去,还是报同样的错误

这个是缺包造成的,可以按照笨方法都添加进去。如果有时间,可以在测试,到底是哪个包的问题
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条