搭建spark环境出现的问题

安装系统为centos6.5，节点上安装了python2和python3的版本，但是在使用时提示如下错误

./pyspark
Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Warning: Ignoring non-spark config property: export=PYSPARK_DRIVER_PYTHON=ipython
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/07 17:04:18 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
17/11/07 17:04:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/opt/spark2/python/pyspark/context.py:194: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
  warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")
17/11/07 17:04:21 ERROR PoolWatchThread: Error in trying to obtain a connection. Retrying in 7000ms
java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
    at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
    at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
    at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
    ... 8 more
17/11/07 17:04:28 ERROR PoolWatchThread: Error in trying to obtain a connection. Retrying in 7000ms
java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
    at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
    at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
    at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
    ... 8 more
17/11/07 17:04:35 ERROR PoolWatchThread: Error in trying to obtain a connection. Retrying in 7000ms
java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
    at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
    at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
    at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
    ... 8 more
求指教

metastore看看它的权限，可以尝试删除，提前做好备份

先确保版本兼容，里面显然识别的是Python2，似乎这个不好用，然后jdk支持的是jdk1.8,楼主用的是jdk1.7.
所以先解决这两个问题。建议存在一个版本的Python及安装jdk1.8.卸载1.7，然后在看后面的问题。

einhep 发表于 2017-11-7 21:05
先确保版本兼容，里面显然识别的是Python2，似乎这个不好用，然后jdk支持的是jdk1.8,楼主用的是jdk1.7.
所 ...

已经安装1.8的jdk依然问题存在

einhep 发表于 2017-11-7 21:05
先确保版本兼容，里面显然识别的是Python2，似乎这个不好用，然后jdk支持的是jdk1.8,楼主用的是jdk1.7.
所 ...

[root@elephant bin]# ./pyspark
Python 3.6.3 (default, Nov  9 2017, 11:50:07)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.
[TerminalIPythonApp] WARNING | Unknown error in handling PYTHONSTARTUP file /opt/spark2/python/pyspark/shell.py:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/IPython/core/shellapp.py in _exec_file(self, fname, shell_futures)
    321                                                  self.shell.user_ns,
    322                                                  shell_futures=shell_futures,
--> 323                                                  raise_exceptions=True)
    324         finally:
    325             sys.argv = save_argv

/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py in safe_execfile(self, fname, exit_ignore, raise_exceptions, shell_futures, *where)
   2489                 py3compat.execfile(
   2490                     fname, glob, loc,
-> 2491                     self.compile if shell_futures else None)
   2492             except SystemExit as status:
   2493                 # If the call was made with 0 or None exit status (sys.exit(0)

/usr/local/lib/python3.6/site-packages/IPython/utils/py3compat.py in execfile(fname, glob, loc, compiler)
    184         with open(fname, 'rb') as f:
    185             compiler = compiler or compile
--> 186             exec(compiler(f.read(), fname, 'exec'), glob, loc)
    187
    188     # Refactor print statements in doctests.

/opt/spark2/python/pyspark/shell.py in <module>()
     28 import py4j
     29
---> 30 import pyspark
     31 from pyspark.context import SparkContext
     32 from pyspark.sql import SparkSession, SQLContext

/opt/spark2/python/pyspark/__init__.py in <module>()
     42
     43 from pyspark.conf import SparkConf
---> 44 from pyspark.context import SparkContext
     45 from pyspark.rdd import RDD
     46 from pyspark.files import SparkFiles

/opt/spark2/python/pyspark/context.py in <module>()
     38     PairDeserializer, AutoBatchedSerializer, NoOpSerializer
     39 from pyspark.storagelevel import StorageLevel
---> 40 from pyspark.rdd import RDD, _load_from_socket, ignore_unicode_prefix
     41 from pyspark.traceback_utils import CallSite, first_spark_call
     42 from pyspark.status import StatusTracker

/opt/spark2/python/pyspark/rdd.py in <module>()
     51 from pyspark.shuffle import Aggregator, ExternalMerger, \
     52     get_used_memory, ExternalSorter, ExternalGroupBy
---> 53 from pyspark.traceback_utils import SCCallSiteSync
     54
     55

/opt/spark2/python/pyspark/traceback_utils.py in <module>()
     21
     22
---> 23 CallSite = namedtuple("CallSite", "function file linenum")
     24
     25

/usr/local/lib/python3.6/collections/__init__.py in namedtuple(*args, **kwargs)
    391
    392     def namedtuple(*args, **kwargs):
--> 393         cls = _old_namedtuple(*args, **kwargs)
    394         return _hack_namedtuple(cls)
    395

TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'

In [1]:

smfswxj 发表于 2017-11-9 11:59
[root@elephant bin]# ./pyspark
Python 3.6.3 (default, Nov  9 2017, 11:50:07)
Type 'copyright', ...

版本兼容问题，不支持Python3.6.3可以按照下面方法操作

spark不支持python3.6
不用卸载python，用anaconda的环境切换就行了
1. 先创建一个新的开发环境: conda create -n my_new_env_python35
2. 激活这个新的开发环境: activate my_new_env_python35
3. 在这个新的开发环境中安装python 3.5: conda install python=3.5
这时python3.5版本的解释器就算是安装完成了，目录为：/home/username/.conda/envs/my_new_env_python35

我觉得没必要较劲，有些浪费时间，之前在首次安装时输入pyspark出现过三个关键词缺失的错误，这里有写解决方法http://blog.csdn.net/sisteryaya/article/details/68945705 。但这次是因为输入：sc.parallelize(range(100),8).take(5) 时，又出现这个错误：
TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'

        你可以按上面的那个方法做，在改变一下环境变量就可以了；或者，直接安装python3.5，但是它的库比较多，在windows上安装过一次，这次在虚拟机上使用，个人觉得装Anaconda就可以了，所有版本在这里：https://repo.continuum.io/archive/
python3.5的版本是这个  Anaconda3-4.2.0-Linux-x86_64.sh