AWS EMR S3DistCp：auxService：mapreduce_shuffle不存在

我通過SSH連接到AWS EMR v5.4.0實例，我想調用s3distcp。 This link演示瞭如何建立電子病歷的步驟來稱呼它，但是當我運行它，我得到以下錯誤：AWS EMR S3DistCp：auxService：mapreduce_shuffle不存在

Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) 
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) 
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155) 
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)

我跟着the instructions here，但它仍然沒有奏效。

來源

2017-04-18 Mark J Miller

原來我需要restart the yarn nodemanager service配置mapreduce_shuffle後：

$ initctl list | grep yarn 
hadoop-yarn-resourcemanager start/running, process 1256 
hadoop-yarn-proxyserver start/running, process 702 
hadoop-yarn-nodemanager start/running, process 896 
$ sudo stop hadoop-yarn-nodemanager 
$ sudo start hadoop-yarn-nodemanager

此外，如果它有助於yarn-site.xml文件位於：/etc/hadoop/conf/yarn-site.xml。它已經有了一個條目yarn.nodemanager.aux-services但mapreduce_shuffle沒有配置：

<property> 
    <name>yarn.nodemanager.aux-services</name> 
    <value>spark_shuffle,</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> 
    <value>org.apache.spark.network.yarn.YarnShuffleService</value> 
</property>

所以我添加這樣的：

<property> 
    <name>yarn.nodemanager.aux-services</name> 
    <value>spark_shuffle,mapreduce_shuffle</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> 
    <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> 
    <value>org.apache.spark.network.yarn.YarnShuffleService</value> 
</property>

來源

2017-04-18 17:48:03

AWS EMR S3DistCp：auxService：mapreduce_shuffle不存在

回答

相關問題