1
我開發了一個使用SparkSQL和SparkStreaming的獨立Spark Spark應用程序。如何打包Spark Spark應用程序
這在Eclipse中配置爲火花正常工作。
我是maven的新手。
要使用maven打包該應用程序，我也跟着下面的教程
http://ryancompton.net/2014/05/19/sample-pomxml-to-build-scala--jar-with-dependenciesjar/
但結束了以下錯誤。
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:104)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.apache.maven.plugin.PluginResolutionException: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:204)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.createPluginRealm(DefaultMavenPluginManager.java:359)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.setupPluginRealm(DefaultMavenPluginManager.java:327)
at org.apache.maven.plugin.DefaultBuildPluginManager.getPluginRealm(DefaultBuildPluginManager.java:229)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:100)
... 20 more
Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:292)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:317)
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:192)
... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:384)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:231)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
... 26 more
Caused by: org.apache.maven.model.resolution.UnresolvableModelException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:170)
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:219)
at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:898)
at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:750)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:309)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:375)
... 31 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:166)
... 36 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1016)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1004)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:725)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.maven.wagon.TransferFailedException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:935)
at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:660)
... 4 more
Caused by: org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:138)
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:365)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.maven.wagon.providers.http.httpclient.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.maven.wagon.providers.http.httpclient.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:756)
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:854)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.maven.wagon.providers.http.httpclient.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
... 18 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
請提供一個關於如何打包我的應用程序並在羣集中運行的解決方案。
你可能改換了一下。我們在這裏提供幫助，而不是解決方案:) – 2014-11-24 11:34:56
您的settings.xml中配置了Maven存儲庫代理嗎？ （你可以在'〜/ .m2/settings.xml'找到它）。它看起來像Maven試圖從代理而不是從Maven Central存儲庫獲取依賴關係。查看例外：'HttpHostConnectException：連接到172.20.181.136:8080 [/172.20.181.136]失敗：連接被拒絕。 – smola 2014-11-24 13:56:53
感謝您的回覆。設置代理設置後，也存在錯誤。我在上面的鏈接中使用了xml文件。它顯示錯誤在該行 斯卡拉 - 編譯 - 第一 過程資源 附加源 編譯 –
2014-11-27 06:13:49