spark安装总结

折腾了一个多星期，发了N个帖子求助和查资料，终于将spark的Standalone安装解决了，总结如下：主要分Standalone，基于文件系统的Standalone，基于zookeeper 的Standalone HA，以下为spark-env.sh配置：

1. Standalone
export SPARK_MASTER_ID=yun01-nn-01
export SPARK_MASTER_PORT=7077
export SPARK_WORK_CORES=1
export SPARK_WORK_INSTANCES=1
export SPARK_WORK_MEMORY=1g

2. 基于文件系统的Standalone
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=FILESYSTEM -Dspark.deploy.recoveryDirectory=/application/hadoop/spark/recovery"

export SPARK_MASTER_ID=yun01-nn-01
export SPARK_MASTER_PORT=7077
export SPARK_WORK_CORES=1
export SPARK_WORK_INSTANCES=1
export SPARK_WORK_MEMORY=1g

3.基于zookeeper 的Standalone HA
export SPARK_WORK_CORES=1
export SPARK_WORK_INSTANCES=1
export SPARK_WORK_MEMORY=1g
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=yun01-nn-02:2181,yun01-dn-01:2181,yun01-dn-02:2181 -Dspark.deploy.zookeeper.dir=/spark"