Hadoop MapReduce Java實現中的減速器

我正在Hadoop MapReduce框架中編寫Java實現程序。我正在寫一個名爲CombinePatternReduce.class的類。爲了在Eclipse中調試減速機，我寫了main()功能如下：Hadoop MapReduce Java實現中的減速器

@SuppressWarnings("unchecked") 
public static void main(String[] args) throws IOException, InterruptedException{ 
    Text key = new Text("key2:::key1:::_ performs better than _"); 
    IntWritable count5 = new IntWritable(5); 
    IntWritable count3 = new IntWritable(3); 
    IntWritable count8 = new IntWritable(8); 
    List<IntWritable> values = new ArrayList<IntWritable>(); 
    values.add(count5); 
    values.add(count3); 
    values.add(count8); 
    CombinePatternReduce reducer = new CombinePatternReduce(); 
    Context dcontext = new DebugTools.DebugReducerContext<Text, IntWritable, KeyPairWritableComparable, WrapperDoubleOrPatternWithWeightWritable>(reducer, key, count3); // here is the problem 
    reducer.reduce(key, values, dcontext);  
}

的DebugTools.DebugReducerContext是我寫盡調試過程中容易執行的一類，它是如下：

public static class DebugReducerContext<KIN, VIN, KOUT, VOUT> extends Reducer<KIN, VIN, KOUT, VOUT>.Context { 
    DebugTools dtools = new DebugTools(); 
    DataOutput out = dtools.new DebugDataOutputStream(System.out); 

    public DebugReducerContext(Reducer<KIN, VIN, KOUT, VOUT> reducer, Class<KIN> keyClass, Class<VIN> valueClass) throws IOException, InterruptedException{ 
     reducer.super(new Configuration(), new TaskAttemptID(), new DebugRawKeyValueIterator(), null, null, 
       null, null, null, null, keyClass, valueClass); 
    } 

    @Override 
    public void write(Object key, Object value) throws IOException, InterruptedException { 
     writeKeyValue(key, value, out); 
    } 

    @Override 
    public void setStatus(String status) { 
     System.err.println(status); 
    } 
}

的問題是在代碼，即main()第一部分。當我寫

Context dcontext = new DebugTools.DebugReducerContext<Text, IntWritable, KeyPairWritableComparable, WrapperDoubleOrPatternWithWeightWritable>(reducer, key, count3);

有是

The constructor DebugTools.DebugReducerContext<Text,IntWritable,KeyPairWritableComparable,WrapperDoubleOrPatternWithWeightWritable>(CombinePatternReduce, Text, IntWritable) is undefined.

當我寫

Context dcontext = new DebugTools.DebugReducerContext<Text, IntWritable, KeyPairWritableComparable, WrapperDoubleOrPatternWithWeightWritable>(reducer, key, values);

有一個錯誤，

The constructor DebugTools.DebugReducerContext<Text,IntWritable,KeyPairWritableComparable,WrapperDoubleOrPatternWithWeightWritable>(CombinePatternReduce, Text, List<IntWritable>) is undefined.

由於Reducer.Context的文件是錯誤的

public Reducer.Context(Configuration conf, 
         TaskAttemptID taskid, 
         RawKeyValueIterator input, 
         Counter inputKeyCounter, 
         Counter inputValueCounter, 
         RecordWriter<KEYOUT,VALUEOUT> output, 
         OutputCommitter committer, 
         StatusReporter reporter, 
         RawComparator<KEYIN> comparator, 
         Class<KEYIN> keyClass, 
         Class<VALUEIN> valueClass) 
       throws IOException, 
         InterruptedException

我需要在Class<KEYIN> keyClass和Class<VALUEIN> valueClass通過。那麼如何編寫main（）函數（特別是帶有錯誤的句子）來調試reducer類呢？

來源

2012-10-28 Denzel

如果你想單元測試你的邏輯，使用MRUnit。如果您有輸入，請使用localrunner。沒有必要構建自己的上下文。 –

這是非常清楚的類的構造函數需要3個參數。 Reducer的一個實例，鍵的類和值的類。

，而不是實際傳遞鍵和值。您需要提供鏈接到類別

Context dcontext = new DebugTools.DebugReducerContext<Text, IntWritable, KeyPairWritableComparable, WrapperDoubleOrPatternWithWeightWritable>(reducer, Text.class, IntWritable.class);

本質上這是重申上下文應該能夠處理以減少的值的類型。

來源

2012-11-08 14:44:23 Bryan

Hadoop MapReduce Java實現中的減速器

回答

相關問題