亞馬遜EMR - 等待步驟

Question

我在寫一個運行aws emr命令（aws emr版本1.5.2）的bash腳本。

如何讓我的腳本等到emr作業完成後再繼續？ --wait-for-steps選項現在已折舊。

通過jq我得到這個，但它只是似乎是一個錯誤的做法：

STEP_STATUS_STATE=$(aws emr list-steps --cluster-id ${CLUSTER_ID} | jq '.Steps[0].Status.State' | tr -d '"') 
while [[ ${STEP_STATUS_STATE} == PENDING ]] || [[ ${STEP_STATUS_STATE} == RUNNING ]]; do 
    STEP_STATUS_STATE=$(aws emr list-steps --cluster-id ${CLUSTER_ID} | jq '.Steps[0].Status.State' | tr -d '"') 
    echo $(date) ${STEP_STATUS_STATE} 
    sleep 10 
done 

Answer 1

我使用AWS的Java API要等到作業完成後如下圖所示。希望這有助於

public static final List<JobFlowExecutionState> DONE_STATES = Arrays 
     .asList(new JobFlowExecutionState[] { 
       JobFlowExecutionState.COMPLETED, 
       JobFlowExecutionState.FAILED, 
       JobFlowExecutionState.TERMINATED }); 

...

public static boolean isDone(String value) { 
    JobFlowExecutionState state = JobFlowExecutionState.fromValue(value); 
    return Constants.DONE_STATES.contains(state); 
} 

    . 
    . 
    STATUS_LOOP: while (true) { 
     DescribeJobFlowsRequest desc = new DescribeJobFlowsRequest(
       Arrays.asList(new String[] { result.getJobFlowId() })); 
     DescribeJobFlowsResult descResult = emr.describeJobFlows(desc); 
     for (JobFlowDetail detail : descResult.getJobFlows()) { 
      String state = detail.getExecutionStatusDetail().getState(); 
      if (isDone(state)) { 
       logger.info("Job " + state + ": " + detail.toString()); 

       if(loadToDailyDB && state.equalsIgnoreCase("COMPLETED")) 
       { 

        //Do something 
       } 
       if(!state.equalsIgnoreCase("COMPLETED")) 
       { 

       } 

       break STATUS_LOOP; 
      } else if (!lastState.equals(state)) { 
       lastState = state; 
       logger.info("Job " + state + " at " 
         + new Date().toString()); 
      } 
     } 
     Thread.sleep(75000);