Spring Boot YARN無法在Hadoop 2.8.0客戶端上運行客戶端無法訪問DataNode

我試圖運行Spring Boot YARN示例（Windows上爲https://spring.io/guides/gs/yarn-basic/）。在application.yml我改變了fsUri和resourceManagerHost指向我的虛擬主機192.168...。但是，當我試圖運行的應用程序Exceprion出現：Spring Boot YARN無法在Hadoop 2.8.0客戶端上運行客戶端無法訪問DataNode

DFSClient: Exception in createBlockOutputStream 
java.net.ConnectException: Connection timed out: no further information 
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) 
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) 
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1508) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1284) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449) 
[2017-05-27 19:59:49.570] boot - 7728 INFO [Thread-5] --- DFSClient: Abandoning BP-646365587-10.0.2.15-1495898351938:blk_1073741830_1006 
[2017-05-27 19:59:49.602] boot - 7728 INFO [Thread-5] --- DFSClient: Excluding datanode DatanodeInfoWithStorage[10.0.2.15:50010,DS-f909ec7a-8374-4cdd-9cfc-0e778810d98c,DISK] 
[2017-05-27 19:59:49.647] boot - 7728 WARN [Thread-5] --- DFSClient: DataStreamer Exception 
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /app/gs-yarn-basic/gs-yarn-basic-container-0.1.0.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

這意味着，數據管理部是不是從我的主機訪問。出於這個原因，我加入到hdfs-site.xml中

<property> 
    <name>dfs.client.use.datanode.hostname</name> 
    <value>true</value> 
    <description>Whether clients should use datanode hostnames when 
    connecting to datanodes. 
    </description> 
</property>

但它仍然會拋出該異常。

我的虛擬機上運行Hadoop 2.8.0。這是conf。文件：

核心的site.xml

<configuration> 
    <property> 
     <name>fs.defaultFS</name> 
     <value>hdfs://0.0.0.0:9000</value> 
    </property> 

</configuration>

HDFS-site.xml中

<configuration> 
     <property> 
      <name>dfs.replication</name> 
      <value>1</value> 
     </property> 
     <property> 
      <name>dfs.namenode.name.dir</name> 
      <value>/usr/local/hadoop/hadoop-2.8.0/data/namenode</value> 
     </property> 

     <property> 
      <name>dfs.datanode.data.dir</name> 
      <value>/usr/local/hadoop/hadoop-2.8.0/data/datanode</value> 
     </property> 

     <property> 
      <name>dfs.permissions.enabled</name> 
      <value>false</value> 
     </property> 

     <property> 
      <name>dfs.client.use.datanode.hostname</name> 
      <value>true</value> 
      <description>Whether clients should use datanode hostnames when 
       connecting to datanodes. 
      </description> 
     </property> 
    </configuration>

mapred-site.xml中

<configuration>  
    <property> 
     <name>mapreduce.framework.name</name> 
     <value>yarn</value> 
    </property> 
</configuration>

紗的site.xml

<configuration> 
    <property> 
     <name>yarn.nodemanager.aux-services</name> 
     <value>mapreduce_shuffle</value> 
    </property> 
    <property> 
     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
     <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
    </property> 
    <property> 
     <name>yarn.scheduler.maximum-allocation-mb</name> 
     <value>8192</value> 
    </property> 
     <property> 
     <name>yarn.nodemanager.resource.memory-mb</name> 
     <value>8192</value> 
    </property> 
    <property> 
     <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per- 
      disk-percentage</name> 
     <value>99</value> 
    </property>  
</configuration>

來源

2017-05-27 Markiza

爲什麼在'core-site.xml中放入'0.0.0.0：9000'？這應該是IP或主機名。 –

@RameshMaharjan，它工作後改爲IP，謝謝 – Markiza

您的core-site.xml應指向Namenode地址，但其當前指向0.0.0.0這意味着本地計算機上的所有地址。這會造成模糊的結果，因爲每臺機器都應被視爲Namenode。

Namenode應該只在hadoop集羣中有一個。

用Namenode的ip或hostname替換0.0.0.0應解決您所面臨的問題。

來源

2017-06-01 15:49:46

它真的有效！謝謝！:) – Markiza

你認爲它不應該被刪除？ – Markiza

這並不糟糕 – Markiza

將core-site.xml中的0.0.0.0:9000更改爲[VM的IP]：9000後，Spring連接到YARN。感謝@RameshMaharjan

來源

2017-05-30 15:48:09 Markiza

@RameshMaharjan，Ohhh，對不起:)謝謝！ – Markiza

@RameshMaharjan，當然:) – Markiza

Spring Boot YARN無法在Hadoop 2.8.0客戶端上運行客戶端無法訪問DataNode

回答

相關問題