火花工人不會綁定到主人

啓動我的火花工人，我得到了一個錯誤，這可能與從機聯繫主機有關。但我不確定。火花工人不會綁定到主人

6/02/12 23:47:13 INFO Utils: Successfully started service 'sparkWorker' on port 38019. 
16/02/12 23:47:13 INFO Worker: Starting Spark worker 192.168.0.38:38019 with 8 cores, 26.5 GB RAM 
16/02/12 23:47:13 INFO Worker: Running Spark version 1.6.0 
16/02/12 23:47:13 INFO Worker: Spark home: /home/romain/spark-1.6.0-bin-hadoop2.6 
16/02/12 23:47:13 INFO Utils: Successfully started service 'WorkerUI' on port 8081. 
16/02/12 23:47:13 INFO WorkerWebUI: Started WorkerWebUI at http://192.168.0.38:8081 
16/02/12 23:47:13 INFO Worker: Connecting to master 192.168.0.39:7078... 
16/02/12 23:47:13 WARN Worker: Failed to connect to master 192.168.0.39:7078 
java.io.IOException: Failed to connect to /192.168.0.39:7078 
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216) 
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167) 
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200) 
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) 
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.net.ConnectException: Connection refused: /192.168.0.39:7078 
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) 
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) 
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) 
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) 
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) 
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    ... 1 more

雖然在主我看到它運行起來：

16/02/12 23:30:30 WARN Utils: Your hostname, pl resolves to a loopback address: 127.0.1.1; using 192.168.0.39 instead (on interface eth0) 
16/02/12 23:30:30 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
16/02/12 23:30:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/02/12 23:30:31 INFO SecurityManager: Changing view acls to: romain 
16/02/12 23:30:31 INFO SecurityManager: Changing modify acls to: romain 
16/02/12 23:30:31 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(romain); users with modify permissions: Set(romain) 
16/02/12 23:30:31 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078. 
16/02/12 23:30:31 INFO Utils: Successfully started service 'sparkMaster' on port 7078. 
16/02/12 23:30:31 INFO Master: Starting Spark master at spark://pl:7078 
16/02/12 23:30:31 INFO Master: Running Spark version 1.6.0 
16/02/12 23:30:32 INFO Utils: Successfully started service 'MasterUI' on port 3094. 
16/02/12 23:30:32 INFO MasterWebUI: Started MasterWebUI at http://192.168.0.39:3094 
16/02/12 23:30:32 WARN Utils: Service could not bind on port 6066. Attempting port 6067. 
16/02/12 23:30:32 INFO Utils: Successfully started service on port 6067. 
16/02/12 23:30:32 INFO StandaloneRestServer: Started REST server for submitting applications on port 6067 
16/02/12 23:30:32 INFO Master: I have been elected leader! New state: ALIVE

要通過博客和網頁現在看來，這是可能的，我們需要一個安全的網絡（我沒有安裝密碼的ssh密鑰 - 但對於「romain」用戶：在哪個用戶是火花啓動？命令行用戶，我猜）。

我應該檢查網絡上的東西嗎？從這個頁面： Spark worker can not connect to Master 我想：

telnet 192.168.0.39 
Trying 192.168.0.39... 
telnet: Unable to connect to remote host: Connection refused

但ping成功：

[email protected]:~/spark-1.6.0-bin-hadoop2.6$ ping 192.168.0.39 
PING 192.168.0.39 (192.168.0.39) 56(84) bytes of data. 
64 bytes from 192.168.0.39: icmp_seq=1 ttl=64 time=0.233 ms 
64 bytes from 192.168.0.39: icmp_seq=2 ttl=64 time=0.185 ms 
^C 
--- 192.168.0.39 ping statistics --- 
2 packets transmitted, 2 received, 0% packet loss, time 1000ms 
rtt min/avg/max/mdev = 0.185/0.209/0.233/0.024 ms

和我有密碼的ssh連接：

$ ssh 192.168.0.39 
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-49-generic x86_64) 
$

應該怎樣做才能使連接可能嗎？

來源

2016-02-12 romain jouin

似乎與特定的端口號問題......也許你可以嘗試關閉本地操作系統防火牆。 – Sumit

我在ubuntu 14.04上，有沒有默認的操作系統防火牆？ –

嘗試在master和worker上運行這個命令：sudo iptables -A INPUT -s 0/0 -p tcp -m tcp --dport 7078 -j ACCEPT'和'sudo/sbin/service iptables save'，看看它是否有效。 – Sumit

通過設置SPARK_LOCAL_IP = 127.0.0.1變量，我能夠讓我的火花工人工作。

你可以將它定義爲局部ENV bash的變量在〜/ .bashrc
可以使$ SPARK_HOME/conf目錄/火花env.sh.template的副本「的conf /火花ENV。 sh'並在那裏定義它。

在羣集環境中，您最好將其作爲本地IP地址。因此，您將能夠看到工人節點UI。

來源

2017-10-07 11:48:58 Sampath

火花工人不會綁定到主人

回答

相關問題