sparkSQL的SBT依賴

我開始學習spark sql我在sbt中使用下列依賴關係。我收到錯誤sparkSQL的SBT依賴

name := "sparkLearning" 

version := "1.0" 

scalaVersion := "2.11.8" 

val sparkVersion = "1.6.1" 
val sqlVersion = "1.3.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" % "spark-sql" % sqlVersion 
)

我收到錯誤。

Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ... 
[info] Resolving org.scala-sbt#completion;0.13.15 ... 
[info] Resolving org.scala-sbt#control;0.13.15 ... 
[info] Resolving org.scala-sbt#sbt;0.13.15 ... 
[info] Resolving org.scala-sbt#run;0.13.15 ... 
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ... 
[info] Resolving org.scala-sbt#test-interface;1.0 ... 
[info] Resolving com.jcraft#jsch;0.1.50 ... 
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ... 
[info] Resolving jline#jline;2.14.3 ... 
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ... 
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ... 
[info] Resolving org.scala-sbt#logic;0.13.15 ... 
[info] Resolving org.scala-sbt#main-settings;0.13.15 ... 
[trace] Stack trace suppressed: run 'last *:update' for the full output. 
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output. 
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found 
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found 
[error] Total time: 15 s, completed 27-Jul-2017 15:29:52 
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0

請讓我知道如何解決這個問題。

來源

2017-07-27 Srinivas

正確的形式爲您SBT文件是

name := "sparkLearning" 

version := "1.0" 

scalaVersion := "2.11.8" 

val sparkVersion = "1.6.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.10" % sparkVersion, 
    "org.apache.spark" % "spark-sql_2.10" % sparkVersion 
)

我會建議你使用這應該是使用Scala 2.11.8

name := "sparkLearning" 

version := "1.0" 

scalaVersion := "2.11.8" 

val sparkVersion = "2.2.0" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" %% "spark-sql" % sparkVersion 
)

來源

2017-07-27 14:44:37

感謝您的幫助兼容最新版本的火花。 – Srinivas

感謝您的接受:)你也可以upvote如果它幫助你 –

如果你打算使用spark-submit運行你的火花程序，你需要添加提供給火花依賴關係： val sparkVersion =「2.2.0 「 ... 」org.apache.spark「%%」spark-core「％sparkVersion％Provided，」org.apache.spark「%%」spark-sql「％sparkVersion％Provided， ... –

sparkSQL的SBT依賴

回答

相關問題