分享

用bigdatabench测试出现以下问题

执行run_MicroBenchmarks.sh的时候报错:
java.lang.ClassNotFoundException: cn.ac.ict.bigdatabench.Sort
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
conf.properties内容如下:

##
export BigdataBench_Home=/usr/local/bigdatabench/BigDataBench_V3.2.5_Spark
##JAVA
export JAVA_HOME=/usr/local/java/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
export JRE_HOME=$JAVA_HOME/jre
##HADOOP
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=$HADOOP_HOME/conf
export HADOOP_LIB=${HADOOP_HOME}/lib
export CLASSPATH=$HADOOP_HOME/hadoop-core-*.jar:$CLASSPATH
##SPARK
export SCALA_HOME=/usr/local/scala/scala-2.12.0
export PATH=$PATH:$SCALA_HOME/bin
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
export JAR_FILE=$BigdataBench_Home/JAR_FILE/bigdatabench-spark_1.3.0-hadoop_1.0.4.jar

已有(7)人评论

跳转到指定楼层
einhep 发表于 2017-3-1 19:53:06
如何运行的,贴出截图来
回复

使用道具 举报

desehawk 发表于 2017-3-2 09:20:26
配置没看出问题,看看权限。
特别是
export JAR_FILE=$BigdataBench_Home/JAR_FILE/bigdatabench-spark_1.3.0-hadoop_1.0.4.jar


回复

使用道具 举报

lsy1996 发表于 2017-3-2 15:31:04
desehawk 发表于 2017-3-2 09:20
配置没看出问题,看看权限。
特别是
export JAR_FILE=$BigdataBench_Home/JAR_FILE/bigdatabench-spark_1 ...

请问看什么的权限?
回复

使用道具 举报

lsy1996 发表于 2017-3-2 15:34:48
einhep 发表于 2017-3-1 19:53
如何运行的,贴出截图来

运行的截图如下
38.PNG
回复

使用道具 举报

lsy1996 发表于 2017-3-2 15:36:10
run_MicroBenchmarks.sh的内容如下:
#!/bin/bash
##

algorithm=( Sort Grep Wordcount)
if [ -n "$1" ]; then
  choice=$1
else
  echo "Please select a number to choose the corresponding Workload algorithm"
  echo "1. ${algorithm[0]} Workload"
  echo "2. ${algorithm[1]} Workload"
  echo "3. ${algorithm[2]} Workload"

  read -p "Enter your choice : " choice
fi

echo "ok. You chose $choice and we'll use ${algorithm[$choice-1]} Workload"
Workloadtype=${algorithm[$choice-1]}

if [ "x$Workloadtype" == "xSort" ]; then
/usr/local/hadoop/bin/hadoop fs -rmr /spark-sort-result
/usr/local/spark/bin/spark-submit --class cn.ac.ict.bigdatabench.Sort $JAR_FILE /data-MicroBenchmarks /spark-sort-result

elif [ "x$Workloadtype" == "xGrep" ]; then
echo "the keyword to filter the text K:"
read K
/usr/local/hadoop/bin/hadoop fs -rmr /spark-grep-result
/usr/local/spark/bin/spark-submit --class cn.ac.ict.bigdatabench.Grep $JAR_FILE /data-MicroBenchmarks ${K} /spark-grep-result

elif [ "x$Workloadtype" == "xWordcount" ]; then
/usr/local/hadoop/bin/hadoop fs -rmr /spark-wordcount-result
/usr/local/spark/bin/spark-submit --class cn.ac.ict.bigdatabench.WordCount $JAR_FILE /data-MicroBenchmarks /spark-wordcount-result
回复

使用道具 举报

einhep 发表于 2017-3-3 07:49:50
lsy1996 发表于 2017-3-2 15:36
run_MicroBenchmarks.sh的内容如下:
#!/bin/bash
##

第一去掉sudo
第二看下/spark-sort-result看看这个目录是否存在


回复

使用道具 举报

lsy1996 发表于 2017-3-9 11:05:10
这个问题已经解决,是设置的环境变量在脚本里没有用,我直接代入的路径就可以跑了。。
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条