Spark2.1.0不兼容的版本，傑克遜2.7.6

Question

我試圖運行的IntelliJ一個簡單的例子火花，但我得到這樣的錯誤：

Exception in thread "main" java.lang.ExceptionInInitializerError 
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701) 
at org.apache.spark.SparkContext.textFile(SparkContext.scala:819) 
at spark.test$.main(test.scala:19) 
at spark.test.main(test.scala) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.6 
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) 
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) 
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730) 
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) 
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala) 

我嘗試更新我的傑克遜的依賴，但它似乎不工作，我這樣做：

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 

但它仍然出現相同的錯誤信息，可以有人幫我修復錯誤？

這裏是火花示例代碼：

object test { 
def main(args: Array[String]): Unit = { 
    if (args.length < 1) { 
     System.err.println("Usage: <file>") 
     System.exit(1) 
    } 

    val conf = new SparkConf() 
    val sc = new SparkContext("local","wordcount",conf) 
    val line = sc.textFile(args(0)) 

    line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println) 

    sc.stop() 
    } 
} 

這裏是我的built.sbt：

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0" 

Answer 1

星火2.1.0包含com.fasterxml.jackson.core作爲傳遞依賴。所以，我們不需要包含在libraryDependencies中。

但是，如果你想添加不同的com.fasterxml.jackson.core依賴關係的版本，那麼你必須重寫它們。就像這樣：

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0" 

因此，改變你的build.sbt像上面的，並預期它會奏效。

我希望它有幫助！

Answer 2

僅供參考。對於我的情況，我在應用中使用spark和kafka-streams，而kafka-streams使用com.fasterxml.jackson.core 2.8.5。下面添加exclude固定問題

（gradle這個）

compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){ 
    exclude group:"com.fasterxml.jackson.core" 
}