SparkException：任務不可序列

我跑我的程序上星火但SparkException拋出：SparkException：任務不可序列

Exception in thread "main" org.apache.spark.SparkException: Task not serializable 
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166) 
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158) 
    at org.apache.spark.SparkContext.clean(SparkContext.scala:1242) 
    at org.apache.spark.rdd.RDD.flatMap(RDD.scala:277) 
    at com.ynu.App$.main(App.scala:42) 
    at com.ynu.App.main(App.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
    Caused by: java.io.NotSerializableException: org.apache.spark.SparkContext 
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) 
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) 
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) 
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) 
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) 
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) 
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73) 
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164) 
... 12 more

有人能告訴我如何解決它？謝謝。

來源

2014-12-03 fanhk

您是否擴展'App'並直接在對象中編寫代碼？如果是這樣，你可以使用'main'方法嗎？ – zsxwing 2014-12-04 06:35:04

是的我擴展應用程序，但我的程序運行在集羣上，爲什麼我應該使用主要方法？ – fanhk 2014-12-06 01:13:48

'extends App'可能會使一些Closure函數存儲對SparkContext或其他不可序列化實例的引用。 – zsxwing 2014-12-06 02:35:54

對此的最佳答案是在您使用擴展應用程序Scala類它爲我工作。

來源

2015-02-26 07:08:10

SparkException：任務不可序列

回答

相關問題