0
我正在寫MRjob，並且想要在基於密鑰的分區上對reducer輸出進行分區。 我正在使用這些選項並獲得以下錯誤。如何使用keyfieldbasedpartitioner？我需要爲此下載一些東西嗎？ MRJOB是用python編寫的。-partitioner：class not found：org.apache.Hadoop.mapred.lib.KeyFieldBasedPartitioner
Step 1 of 1 failed: Command '['hadoop', 'jar',
'/usr/lib/hadoop-mapreduce/hadoop-streaming.jar', '-files',
'hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/abc.py#parsec_status_error_fedactivity.py,hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/setup-wrapper.sh#setup-wrapper.sh',
'-archives',
'hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/mrjob.tar.gz#mrjob.tar.gz',
'-D', 'mapreduce.job.name=abc', '-D', 'mapreduce.job.reduces=2',
'-D', 'mapreduce.job.split.metainfo.maxsize=-1', '-D',
'mapreduce.map.failures.maxpercent=1', '-D',
'mapreduce.map.java.opts=-Xmx1g', '-D',
'mapreduce.map.memory.mb=2048', '-D',
'mapreduce.output.fileoutputformat.compress=true', '-D',
'mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec',
'-D', 'mapreduce.partition.keypartitioner.options=-k1', '-D',
'mapreduce.reduce.java.opts=-Xmx2g', '-D',
'mapreduce.reduce.memory.mb=3072', '-D',
'mapreduce.reduce.shuffle.input.buffer.percent=0.4', '-D',
'mapreduce.reduce.shuffle.merge.percent=0.4', '-D',
'stream.map.input.ignoreKey=true', '-D',
'stream.num.map.output.key.fields=5', '-libjars',
'/opt/parsec/lib/correctionlayer2.jar', '-partitioner',
'org.apache.Hadoop.mapred.lib.KeyFieldBasedPartitioner', '-input',
'hdfs:////10.134.71.100.1500076800077.gz', '-output',
'hdfs:///20170715', '-mapper', 'sh -ex setup-wrapper.sh python abc.py
--step-num=0 --mapper', '-reducer', 'sh -ex setup-wrapper.sh python abc.py --step-num=0 --reducer']' returned non-zero exit status 256de
here
運行的1步1 ...
-partitioner：未找到類：org.apache.Hadoop.mapred.lib.KeyFieldBasedPartitioner
嘗試-help以獲取更多信息
流式命令失敗！