當你調用spark-submit時,它會爲你自己添加它自己的python文件夾和py4j庫到PYTHONPATH。他們住在spark安裝到的文件夾中,它們不會直接安裝到您正在使用的python的站點包中。例如,如果我pyspark的打印出來的蟒蛇路徑:
$ pyspark
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
17/10/17 15:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/10/17 15:34:04 WARN Utils: Your hostname, rwidmaier resolves to a loopback address: 127.0.0.1; using 10.4.248.126 instead (on interface em1)
17/10/17 15:34:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
/home/rwidmaier/apps/spark/python/pyspark/context.py:195: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")
17/10/17 15:34:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/10/17 15:34:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/10/17 15:34:09 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
____ __
/__/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__/.__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Python version 2.6.6 (r266:84292, Aug 18 2016 15:13:37)
SparkSession available as 'spark'.
>>> import sys
>>> from pprint import pprint
>>> pprint(sys.path)
['',
u'/tmp/spark-0a08efa7-714a-498b-b5e9-bba48684d52d/userFiles-52012e25-d7af-4599-a214-9637141ed4ec',
'/usr/lib64/python2.6/site-packages/deluge-1.3.11-py2.6-linux-x86_64.egg',
'/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip',
'/home/ryan/apps/spark/python',
'/usr/lib64/python26.zip',
'/usr/lib64/python2.6',
'/usr/lib64/python2.6/plat-linux2',
'/usr/lib64/python2.6/lib-tk',
'/usr/lib64/python2.6/lib-old',
'/usr/lib64/python2.6/lib-dynload',
'/usr/lib64/python2.6/site-packages',
'/usr/lib64/python2.6/site-packages/gst-0.10',
'/usr/lib64/python2.6/site-packages/gtk-2.0',
'/usr/lib64/python2.6/site-packages/webkit-1.0',
'/usr/lib/python2.6/site-packages']
您可以在列表中看到它有手動添加:
- /家庭/瑞安/應用/火花/ Python的/ lib目錄/py4j-0.10.4-src.zip
- /家庭/瑞安/應用/火花/蟒蛇
爲了確保解釋總是設置與正確的庫,你只需要設置PYSPARK_PYTHON點到python安裝你會可以使用,然後運行「pyspark」而不是直接運行python。或者,如果您想提供驅動程序,則可以使用spark-submit。
你的'PYTHONPATH'環境變量是什麼? –
@JohnGordon':/ mnt/anaconda/bin' – timbram
python模塊似乎不太可能安裝在'bin'目錄下,所以這可能是你的問題。如何設置PYTHONPATH? –