sbt組件着色來創建在火花上運行的胖罐子

Question

我使用sbt組件來創建一個可以在火花上運行的胖罐子。依賴於grpc-netty。火花上的Guava版本比grpc-netty所要求的要舊，我遇到了這個錯誤：java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument。我能夠通過在spark上將userClassPathFirst設置爲true來解決此問題，但會導致其他錯誤。

糾正我，如果我錯了，但從我的理解，我不應該將userClassPathFirst設置爲true，如果我做正確的陰影。以下是我如何做陰影：

assemblyShadeRules in assembly := Seq(
    ShadeRule.rename("com.google.guava.**" -> "my_conf.@1") 
    .inLibrary("com.google.guava" % "guava" % "20.0") 
    .inLibrary("io.grpc" % "grpc-netty" % "1.1.2") 
) 

libraryDependencies ++= Seq(
    "org.scalaj" %% "scalaj-http" % "2.3.0", 
    "org.json4s" %% "json4s-native" % "3.2.11", 
    "org.json4s" %% "json4s-jackson" % "3.2.11", 
    "org.apache.spark" %% "spark-core" % "2.2.0" % "provided", 
    "org.apache.spark" % "spark-sql_2.11" % "2.2.0" % "provided", 
    "org.clapper" %% "argot" % "1.0.3", 
    "com.typesafe" % "config" % "1.3.1", 
    "com.databricks" %% "spark-csv" % "1.5.0", 
    "org.apache.spark" % "spark-mllib_2.11" % "2.2.0" % "provided", 
    "io.grpc" % "grpc-netty" % "1.1.2", 
    "com.google.guava" % "guava" % "20.0" 
) 

我在做什麼錯在這裏，我該如何解決它？

Answer 1

你快到了。什麼shadeRule所做的就是renames class names，而不是庫名稱：

The main ShadeRule.rename rule is used to rename classes. All references to the renamed classes will also be updated.

事實上，在com.google.guava:guava有與包com.google.guava無類：

$ jar tf ~/Downloads/guava-20.0.jar | sed -e 's:/[^/]*$::' | sort | uniq 
META-INF 
META-INF/maven 
META-INF/maven/com.google.guava 
META-INF/maven/com.google.guava/guava 
com 
com/google 
com/google/common 
com/google/common/annotations 
com/google/common/base 
com/google/common/base/internal 
com/google/common/cache 
com/google/common/collect 
com/google/common/escape 
com/google/common/eventbus 
com/google/common/graph 
com/google/common/hash 
com/google/common/html 
com/google/common/io 
com/google/common/math 
com/google/common/net 
com/google/common/primitives 
com/google/common/reflect 
com/google/common/util 
com/google/common/util/concurrent 
com/google/common/xml 
com/google/thirdparty 
com/google/thirdparty/publicsuffix 

它應該是足夠的明暗規則改成這樣：

assemblyShadeRules in assembly := Seq(
    ShadeRule.rename("com.google.common.**" -> -> "my_conf.@1") 
    .inLibrary("com.google.guava" % "guava" % "20.0") 
    .inLibrary("io.grpc" % "grpc-netty" % "1.1.2") 
) 

所以你不需要改變userClassPathFirst。

而且，可以簡化您的明暗規則是這樣的：

assemblyShadeRules in assembly := Seq(
    ShadeRule.rename("com.google.common.**" -> -> "my_conf.@1").inAll 
) 

由於org.apache.spark依賴性provided，他們將不會被包含在您的罐子，不會被遮蔽（因此火花將使用自己的無陰影版本它在羣集上具有番石榴）。