Hadoop示例安裝後不工作

您好我最近安裝hadoop 2.7.2在分佈式模式下，namenode是hadoop和datanode是hadoop1和hadoop2。當我在bash yarn jar /usr/local/hadoop/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 2 1000，它給我的錯誤信息，如：Hadoop示例安裝後不工作

Number of Maps = 2 
Samples per Map = 1000 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.; Host Details : local host is: "benji/192.168.1.4"; destination host is: "hadoop":9000; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1479) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1412) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) 
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) 
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) 
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) 
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) 
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) 
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) 
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) 
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) 
    at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:278) 
    at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
    at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) 
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) 
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type. 
    at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:99) 
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:498) 
    at com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:193) 
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:2207) 
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:2165) 
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:2295) 
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:2290) 
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200) 
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) 
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) 
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) 
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) 
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) 
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1085) 
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:979)

如果我去做hadoop jar /usr/local/hadoop/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 2 1000，它提供了錯誤信息，如：

Number of Maps = 2 
Samples per Map = 1000 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.; Host Details : local host is: "hadoop/192.168.1.4"; destination host is: "hadoop":9000; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) 
... blabla ...

注意兩個錯誤消息之間的怪異神祕的差異位於本地主機名（一個是benji/192.168.1.4，另一個是hadoop/192.168.1.4）。我做start-dfs.sh,start-yarn.sh之前的yarn jar ...，一切看起來不錯。

如果有人能幫助解決問題，我將非常感激。下面是一些配置文件內容：

/etc/hosts文件（benji是在主計算機上的非hadoop的帳戶）：

192.168.1.4  hadoop benji 

192.168.1.5  hadoop1 

192.168.1.9  hadoop2

/etc/hostname文件：
```
hadoop 
```

~/.ssh/config file：

# hadoop1 
Host hadoop1 
HostName 192.168.1.5 
User hadoop1 
IdentityFile ~/.ssh/hadoopid 

# hadoop2 
Host hadoop2 
HostName 192.168.1.9 
User hadoop2 
IdentityFile ~/.ssh/hadoopid 

# hadoop localhost 
Host localhost 
HostName localhost 
User hadoop 
IdentityFile ~/.ssh/hadoopid 

# hadoop 
Host hadoop 
HostName hadoop 
User hadoop 
IdentityFile ~/.ssh/hadoopid

core-site.xml文件：

<configuration> 

<property> 

    <name>hadoop.tmp.dir</name> 

    <value>file:///usr/local/hadoop/hadoopdata/tmp</value> 

    <description>A base for other temporary directories.</description> 

</property> 

<property> 

<name>fs.defaultFS</name> 

<value>hdfs://hadoop:9000</value> 

</property> 

<property> 

    <name>io.file.buffer.size</name> 

    <value>131072</value> 

</property> 

</configuration>

hdfs-site.xml文件：

<configuration> 

<property> 

<name>dfs.namenode.name.dir</name> 

<value>file:///usr/local/hadoop/hadoopdata/dfs/namenode</value> 

</property> 

<property> 

<name>dfs.datanode.du.reserved</name> 

<value>21474836480</value> 

</property> 

<property> 

<name>dfs.datanode.data.dir</name> 

<value>file:///usr/local/hadoop/hadoopdata/dfs/datanode</value> 

</property> 

<property> 

<name>dfs.replication</name> 

<value>1</value> 

</property> 

<property> 

<name>dfs.namenode.secondary.http-address</name> 

<value>hadoop:9001</value> 

</property> 

<property> 

<name>dfs.webhdfs.enabled</name> 

<value>true</value> 

</property> </configuration>

任何人都可以在這個問題上幫助？謝謝！

UPDATE 1

我想出問題的一部分。我做了jps，發現datanode和namenode沒有運行。在netstat -an | grep 9000和lsof -i :9000之後，我發現另一個進程正在監聽端口9000。在我將fs.defaultFS從hdfs://hadoop:9000改爲hdfs://hadoop:9001的core-site.xml文件和dfs.namenode.secondary.http-address從hadoop:9001改爲hadoop:9002的hdfs-site.xml之後，namenode能夠運行。此更改後protocol-buffer錯誤消息消失。但是根據jps的結果，datanode仍然沒有運行。

數據管理部日誌文件顯示了一些奇怪的事情發生：

... blabla ... 
2016-05-19 12:27:12,157 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop/192.168. 
1.4:9000. Already tried 44 time(s); maxRetries=45 
2016-05-19 12:27:32,158 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to se 
rver: hadoop/192.168.1.4:9000 
... blabla ... 
2016-05-19 13:41:55,382 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM 
2016-05-19 13:41:55,387 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
... blabla ...

我不明白爲什麼數據管理部嘗試連接到名稱節點上的端口9000。

來源

2016-05-23 Fengchong Wang

「/ etc/hosts」中的「127.0.0.1 localhost」在哪裏？ –

我以前評論過它，因爲它沒有在分佈式模式下使用。我剛剛添加了'127.0.0.1 localhost'，但錯誤消息相同。 –

您是否可以對主設備的所有從設備進行無密碼驗證 – gonephishing

您應該安裝在所有的從站配置Hadoop的包裝，只是在名稱節點改變fs.defaultFS到hdfs://hadoop:9001但不是數據節點將產生數據節點嘗試，因爲它在其core-site.xml的既定連接到hdfs://hadoop:900。

來源

2016-05-30 09:50:09 avalanchesiqi

我有'scp'配置文件從主機到從機。錯誤不是因爲這個。 –

Hadoop示例安裝後不工作

回答

相關問題