2016-01-21 99 views
1

我寫了下面的程序。 我已經運行它沒有使用TotalOrderPartitioner,它運行良好。所以我認爲Mapper或Reducer類沒有任何問題。使用TotalOrderPartitioner時遇到錯誤MapReduce

但是,當我包含代碼爲TotalOrderPartitioner即編寫分區文件,然後把它放在分佈式緩存中,我得到以下錯誤:真的很無能的如何去做。

[列車@沙箱TOTALORDERPARTITIONER] $ hadoop的罐子totalorderpart.jar average.AverageJob縣totpart

//縣是輸入目錄和totpart是輸出目錄

16/01/18 04:14:00 INFO input.FileInputFormat: Total input paths to process : 4 16/01/18 04:14:00 INFO partition.InputSampler: Using 6 samples 16/01/18 04:14:00 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 16/01/18 04:14:00 INFO compress.CodecPool: Got brand-new compressor [.deflate] java.io.IOException: wrong key class: org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.Text at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.append(SequenceFile.java:1380) at org.apache.hadoop.mapreduce.lib.partition.InputSampler.writePartitionFile(InputSampler.java:340) at average.AverageJob.run(AverageJob.java:132) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at average.AverageJob.main(AverageJob.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

我的代碼

​​

回答

1

TotalOrderPartitioner不會在Mapper的輸出上運行它的採樣,但是在i輸入數據集。您的輸入格式具有LongWritable作爲鍵和Text作爲值。相反,您試圖調用RandomSampler聲稱您的格式具有Text作爲鍵和Text作爲值。這是一個不匹配它運行時InputSampler認定,因此該消息

wrong key class: org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.Text

含義,它正在試圖找到文本爲重點(根據您的參數化),但它發現LongWritable代替。