2016-04-27 55 views
0

我試圖運行使用aws sdk ruby​​的步驟來使用hadoop的Amazon ElasticMapReduce服務,而我可以創建集羣和步驟,步驟經常失敗,而手動設置時不使用Web界面無法運行程序「-files」(在目錄「。」中):error = 2,沒有這樣的文件或目錄

emr = Aws::EMR::Client.new 
    cluster_id = "*******" 
    resp = emr.add_job_flow_steps({ 
     job_flow_id: cluster_id, # required 
     steps: [ # required 
     { 
      name: "TestStep", # required 
      action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE 
      hadoop_jar_step: { # required 
      jar: 'command-runner.jar', 
      args:[ 
       "-files", 
       "s3://source123/mapper.py,s3://source123/source_reducer.py", 
       "-mapper", 
       "mapper.py", 
       "-reducer", 
       "source_reducer.py", 
       "-input", 
       "s3://source123/input/", 
       "-output", 
       "s3://source123/output/" 
      ] 
      }, 
     }, 
     ], 
    }) 

我得到的錯誤是這樣

Cannot run program "-files" (in directory "."): error=2, No such file or directory

任何線索?

回答

0

似乎加入Hadoop的流就像下面

emr = Aws::EMR::Client.new 
    cluster_id = "*******" 
    resp = emr.add_job_flow_steps({ 
     job_flow_id: cluster_id, # required 
     steps: [ # required 
     { 
      name: "TestStep", # required 
      action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE 
      hadoop_jar_step: { # required 
      jar: 'command-runner.jar', 
      args:[ 
       "hadoop-streaming", 
       "-files", 
       "s3://source123/mapper.py,s3://source123/source_reducer.py", 
       "-mapper", 
       "mapper.py", 
       "-reducer", 
       "source_reducer.py", 
       "-input", 
       "s3://source123/input/", 
       "-output", 
       "s3://source123/output/" 
      ] 
      }, 
     }, 
     ], 
    }) 
相關問題