Apache Spark獨立於匿名UID（無用戶名）

Question

我在OpenShift平臺上啓動Apache Spark從節點。 OpenShift內部啓動泊塢窗圖像作爲匿名用戶（用戶沒有名字，但只是UID）。我得到以下例外

17/07/17 16:46:53 INFO SignalUtils: Registered signal handler for INT 
12 17/07/17 16:46:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
13 Exception in thread "main" java.io.IOException: failure to login 
14  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824) 
15  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) 
16  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) 
17  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) 
18  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) 
19  at scala.Option.getOrElse(Option.scala:121) 
20  at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) 
21  at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221) 
22  at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:714) 
23  at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:696) 
24  at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 
25 Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name 
26  at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71) 
27  at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133) 
28  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
29  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
30  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
31  at java.lang.reflect.Method.invoke(Method.java:497) 
32  at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) 
33  at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) 
34  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) 
35  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) 
36  at java.security.AccessController.doPrivileged(Native Method) 
37  at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) 
38  at javax.security.auth.login.LoginContext.login(LoginContext.java:587) 
39  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799) 
40  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) 
41  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) 
42  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) 
43  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) 
44  at scala.Option.getOrElse(Option.scala:121) 
45  at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) 
46  at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221) 
47  at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:714) 
48  at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:696) 
49  at org.apache.spark.deploy.worker.Worker.main(Worker.scala) 
50 
51  at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856) 
52  at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) 
53  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) 
54  at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) 
55  at java.security.AccessController.doPrivileged(Native Method) 
56  at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) 
57  at javax.security.auth.login.LoginContext.login(LoginContext.java:587) 
58  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799) 
59  ... 10 more 

我試着在spark-default.conf上設置下列屬性仍然沒有用。

spark.eventLog.enabled    false 
spark.ui.enabled     false 
spark.acls.enable     false 
spark.admin.acls     * 
spark.modify.acls     * 
spark.modify.acls.groups   * 
spark.ui.view.acls.groups   * 
spark.ui.enabled     false 

請問您可以幫我解決這個問題。

感謝

納文

Answer 1

（我保持這個答案，因爲它是有用的瞭解nss_wrapper，但是this other answer作品，而無需安裝或用戶nss_wrapper）

星火希望能夠在passwd中查找它的UID。這個集成扭結可以使用nss_wrapper來解決;使用入口點的圖像該解決方案的一個很好的例子可以在這裏找到：

https://github.com/radanalyticsio/openshift-spark/blob/master/scripts/spark/added/entrypoint

# spark likes to be able to lookup a username for the running UID, if 
# no name is present fake it. 
cat /etc/passwd > /tmp/passwd 
echo "$(id -u):x:$(id -u):$(id -g):dynamic uid:$SPARK_HOME:/bin/false" >> /tmp/passwd 

export NSS_WRAPPER_PASSWD=/tmp/passwd 
# NSS_WRAPPER_GROUP must be set for NSS_WRAPPER_PASSWD to be used 
export NSS_WRAPPER_GROUP=/etc/group 

export LD_PRELOAD=libnss_wrapper.so 

exec "$@" 

如果您有興趣，可以在Openshift使用晶圓廠預星火圖像，我建議開始在這裏：

https://github.com/radanalyticsio/openshift-spark

這些圖像的工具爲Radanalytics.io社區項目，已經產生了很多模具中容易打開創造的火花集羣的一部分生成HIFT。您可以瞭解更多關於該項目的位置：

https://radanalytics.io/get-started

Answer 2

這裏是不需要nss_wrapper一種替代方法。

默認情況下，OpenShift容器以匿名用戶標識和組標識0（又名「根」組）運行。首先，設置你的圖像，使得/etc/passwd由組ID 0擁有，並擁有集團的寫訪問，例如這個Dockerfile片段：

RUN chgrp root /etc/passwd && chmod ug+rw /etc/passwd 

然後你就可以在容器啓動時添加以下的邏輯，例如下面的腳本可以被用作ENTRYPOINT：

#!/bin/bash 

myuid=$(id -u) 
mygid=$(id -g) 
uidentry=$(getent passwd $myuid) 

if [ -z "$uidentry" ] ; then 
    # assumes /etc/passwd has root-group (gid 0) ownership 
    echo "$myuid:x:$myuid:$mygid:anonymous uid:/tmp:/bin/false" >> /etc/passwd 
fi 

exec "$@" 

此入口點腳本將自動爲匿名UID一個passwd文件項，以便在需要它不會失敗的工具。

沒有就這個問題和關於OpenShift匿名UID的相關主題一個很好的博客文章： https://blog.openshift.com/jupyter-on-openshift-part-6-running-as-an-assigned-user-id/