2016-09-18 97 views
0

我有星火1.6.1。我將運行火花外殼

export HADOOP_CONF_DIR=/folder/location 

現在,如果我跑火花外殼: $ ./spark-shell --master紗--deploy模式客戶 我得到這種類型的錯誤(相關部分)

$ 16/09/18 15:49:18 INFO impl.TimelineClientImpl: Timeline service address: http://URL:PORT/ws/v1/timeline/ 
16/09/18 15:49:18 INFO client.RMProxy: Connecting to ResourceManager at URL/IP:PORT 
16/09/18 15:49:18 INFO yarn.Client: Requesting a new application from cluster with 9 NodeManagers 
16/09/18 15:49:19 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (14336 MB per container) 
16/09/18 15:49:19 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 
16/09/18 15:49:19 INFO yarn.Client: Setting up container launch context for our AM 
16/09/18 15:49:19 INFO yarn.Client: Setting up the launch environment for our AM container 
16/09/18 15:49:19 INFO yarn.Client: Preparing resources for our AM container 
16/09/18 15:49:19 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 
16/09/18 15:49:19 ERROR spark.SparkContext: Error initializing SparkContext. 
org.apache.hadoop.security.AccessControlException: Permission denied: user=Menmosyne, access=WRITE, inode="/user/Mnemosyne/.sparkStaging/application_1464874056768_0040":hdfs:hdfs:drwxr-xr-x 
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) 
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) 
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) 

然而,當我運行簡單

$ ./spark-shell 

(沒有指定主人)我在屏幕上獲得比平常更多的配置(即它應該加載hadoop文件夾中的配置)。所以如果我沒有說明主人是紗線,我的火花作業是否仍然會提交給紗線羣呢?

回答

1

火花默認主是當地,這意味着應用程序將運行在集羣中的機器本地,而不是。

一般而言(配置單元,mapreduce,spark等)的紗線應用需要創建臨時文件夾來存儲部分數據和/或當前進程配置。通常這個時間的數據被寫入HDFS的用戶在家中(在你的情況/用戶/摩涅莫辛涅)

你的問題是,你的主文件夾是由用戶創建的HDFS和您的用戶摩涅莫辛涅沒有特權寫在它上面。

然後,spark工作無法在啓動應用程序所需的HDFS中創建時間結構。

我的建議是,您更改主文件夾的所有者(每個用戶應該是其主目錄的所有者),並確認所有者擁有對其主目錄的完全訪問權限。

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/FileSystemShell.html#chown

1

我只是修復了這個問題,火花1.6.2和Hadoop集羣2.6.0 1.複製火花組裝1.6.2-hadoop2.6.0.jar從局部到HDFS HDFS://站長:9000 /火花/火花組裝1.6.2-hadoop2.6.0.jar

2 .in spark-defaults.conf添加參數 spark.yarn.jars hdfs:// Master:9000/spark/spark-assembly-1.6.2-hadoop2.6.0.jar

然後運行火花殼--master紗客戶 如果你想運行在紗線模式火花萬物OK

1件事,不啓動在本地模式火花集羣。