Hadoop的錯誤：錯誤啓動的工作，糟糕的輸入路徑：文件不exist.Streaming命令失敗

Question

我運行Hadoop集羣上的MRJob &我收到以下錯誤：

No configs found; falling back on auto-configuration 
Looking for hadoop binary in $PATH... 
Found hadoop binary: /usr/local/hadoop/bin/hadoop 
Using Hadoop version 2.7.3 
Looking for Hadoop streaming jar in /usr/local/hadoop... 
Found Hadoop streaming jar: /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar 
Creating temp directory /tmp/Mr_Jobs.hduser.20170227.030012.446820 
Copying local files to hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/files/... 
Running step 1 of 1... 
    session.id is deprecated. Instead, use dfs.metrics.session-id 
    Initializing JVM Metrics with processName=JobTracker, sessionId= 
    Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
    Cleaning up the staging area file:/app/hadoop/tmp/mapred/staging/hduser1748755362/.staging/job_local1748755362_0001 
    Error launching job , bad input path : File does not exist: /app/hadoop/tmp/mapred/staging/hduser1748755362/.staging/job_local1748755362_0001/files/Mr_Jobs.py#Mr_Jobs.py 
    Streaming Command Failed! 
Attempting to fetch counters from logs... 
Can't fetch history log; missing job ID 
No counters found 
Scanning logs for probable cause of failure... 
Can't fetch history log; missing job ID 
Can't fetch task logs; missing application ID 
Step 1 of 1 failed: Command '['/usr/local/hadoop/bin/hadoop', 'jar', '/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar', '-files', 'hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/files/Mr_Jobs.py#Mr_Jobs.py,hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/files/mrjob.zip#mrjob.zip,hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/files/setup-wrapper.sh#setup-wrapper.sh', '-input', 'hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/files/File.txt', '-output', 'hdfs:///user/hduser/tmp/mrjob/Mr_Jobs.hduser.20170227.030012.446820/output', '-mapper', 'sh -ex setup-wrapper.sh python3 Mr_Jobs.py --step-num=0 --mapper', '-combiner', 'sh -ex setup-wrapper.sh python3 Mr_Jobs.py --step-num=0 --combiner', '-reducer', 'sh -ex setup-wrapper.sh python3 Mr_Jobs.py --step-num=0 --reducer']' returned non-zero exit status 512 

我正在工作通過這個命令：

python3 /home/bhoots21304/Desktop/MrJobs-MR.py -r Hadoop的HDFS：//input3/File.txt

而且第一行說：發現 沒有CONFIGS;迴歸自動配置

我擡頭看了一下。它說應該有/ etc /文件夾中的mrjob.conf文件的名稱，但它不存在於我的文件系統中的任何地方。 我是否需要創建此文件。如果是的話，應該是什麼內容。

我使用在本文件中提及的指令安裝的hadoop：

https://github.com/ev2900/Dev_Notes/blob/master/Hadoop/notes.txt

此外hadoop-env.sh，芯-site.xml中，mapred-site.xml中，HDFS-site.xml中被配置很好，因爲它的工作，如果我只是運行一個簡單的worcount工作（不mRJob的）

Answer 1

你需要指定mrjob蟒蛇-bin和hadoop_streaming_jar（mRJob的使用「sudo的-H PIP3安裝mrjob」安裝）。 CONF。它應該看起來像這樣，取決於罐子的位置。

runners: 
    hadoop: 
     python_bin: python3 
     hadoop_streaming_jar: /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar