deeplearning4j在紗線羣集上失敗

我想使用master = yarn-cluster在羣集上提交我的jar，但會出錯。deeplearning4j在紗線羣集上失敗

java.lang.NoClassDefFoundError: org/nd4j/Nd4jRegistrator 
     at Main.main(Main.java:51) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552) 
Caused by: java.lang.ClassNotFoundException: org.nd4j.Nd4jRegistrator 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

我設置sparkConf這樣，使用克里奧爾語串，因爲我有例外，當我在POM文件沒有它

SparkConf sparkConf = new SparkConf().setMaster(master).setAppName("DL4J Spark Example"); 
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); 
sparkConf.set("spark.kryo.registrator", "org.nd4j.Nd4jRegistrator"); 
JavaSparkContext sc = new JavaSparkContext(sparkConf);

依賴運行：

<dependencies> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>1.6.0</version> 
     </dependency> 

     <dependency> 
      <groupId>org.deeplearning4j</groupId> 
      <artifactId>deeplearning4j-core</artifactId> 
      <version>0.8.0</version> 
     </dependency> 

     <dependency> 
      <groupId>org.deeplearning4j</groupId> 
      <artifactId>dl4j-spark_2.10</artifactId> 
      <version>0.8.0_spark_1</version> 
     </dependency> 

     <dependency> 
      <groupId>org.nd4j</groupId> 
      <artifactId>nd4j-native-platform</artifactId> 
      <version>0.8.0</version> 
     </dependency> 

     <dependency> 
      <groupId>org.nd4j</groupId> 
      <artifactId>nd4j-kryo_2.10</artifactId> 
      <version>0.8.0</version> 
     </dependency> 

     <dependency> 
      <groupId>com.esotericsoftware</groupId> 
      <artifactId>kryo</artifactId> 
      <version>4.0.0</version> 
     </dependency> 
    </dependencies>

無法找到Nd4jRegistrator類，但爲什麼？當我運行本地這工作正常。

來源

2017-06-15 Antosha19955

我猜你沒有正確設置你的uberjar？你需要確保捆綁一切。你的pom整體看起來不錯。

來源

2017-06-15 23:43:09

deeplearning4j在紗線羣集上失敗

回答

相關問題