我的EMR羣集在哪裏

我想在java上創建一個EMR羣集，但是我既不能在EMR羣集列表中找到它，也不能在EC2上看到請求的實例。我的EMR羣集在哪裏

EMR角色確實存在：

[email protected]:~$ aws iam list-roles | grep EMR 
      "RoleName": "EMR_DefaultRole", 
      "Arn": "arn:aws:iam::removed:role/EMR_DefaultRole" 
      "RoleName": "EMR_EC2_DefaultRole", 
      "Arn": "arn:aws:iam::removed:role/EMR_EC2_DefaultRole"

，現在我的java代碼：

AWSCredentials awsCredentials = new BasicAWSCredentials(awsKey, awsKeySecret); 
AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials); 

StepFactory stepFactory = new StepFactory(); 
     StepConfig enabledebugging = new StepConfig() 
       .withName("Enable debugging") 
       .withActionOnFailure("TERMINATE_JOB_FLOW") 
       .withHadoopJarStep(stepFactory.newEnableDebuggingStep()); 

    HadoopJarStepConfig hadoopConfig1 = new HadoopJarStepConfig() 
    .withJar("s3://foo.bucket/hadoop_jar/2015-01-12/foo.jar") 
    .withMainClass("com.strackoverflow.DriverFoo") // optional main class, this can be omitted if jar above has a manifest 
    .withArgs("--input=s3://foo.bucket/logs/,s3://foo.bucket/morelogs/", "--output=s3://foo.bucket/myEMROutput" , "--inputType=text"); // i have custom java code to handle the --input, --output and --inputType parameters 


    StepConfig customStep = new StepConfig("Step1", hadoopConfig1); 

    Collection <StepConfig> steps = new ArrayList<StepConfig>(); 
    { 
     steps.add(enabledebugging); 
     steps.add(customStep); 
    } 


    JobFlowInstancesConfig instancesConfig = new JobFlowInstancesConfig() 
    .withEc2KeyName("fookey") //not fookey.pem 
    .withInstanceCount(2) 
    .withKeepJobFlowAliveWhenNoSteps(false) // on aws example is set to true 
    .withMasterInstanceType("m1.medium") 
    .withSlaveInstanceType("m1.medium"); 


    RunJobFlowRequest request = new RunJobFlowRequest() 
    .withName("java programatic request") 
    .withAmiVersion("3.3.1") 
    .withSteps(steps) // on the amazon example is lunched debug and hive, here is debug and a jar 
    .withLogUri("s3://devel.rui/emr_clusters/pr01/") 
    .withInstances(instancesConfig) 
    .withVisibleToAllUsers(true); 

     RunJobFlowResult result = emr.runJobFlow(request); 

     System.out.println("toString "+ result.toString()); 
     System.out.println("getJobFlowId "+ result.getJobFlowId()); 
     System.out.println("hashCode "+ result.hashCode());

哪裏是我的集羣？我無法在羣集列表中看到它，輸出文件夾未創建，日誌文件夾保持爲空，並且EC2上沒有任何實例可見。

由程序輸出該

toString {JobFlowId: j-2xxxxxxU} 
getJobFlowId j-2xxxxxU 
hashCode -1xxxxx4

我只好按照指令從這裏創建羣集 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/calling-emr-with-java-sdk.html

而這款以創建Java工作 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-common-programming-sample.html

來源

2015-02-10 SQL.injection

在亞馬遜的例子，該區域未配置。

配置區域後，集羣正常啓動。

AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials); 
    emr.setRegion(Region.getRegion(Regions.EU_WEST_1));

來源

2015-02-11 11:48:00

我的EMR羣集在哪裏

回答

相關問題