2
我有KMeans代碼，我的任務是計算加速，我通過在uni集羣中的不同節點數上運行它來完成它。但是是否可以更改映射器和/或縮減器的數量，以便在單節點上運行時檢查加速中的變化。是否有可能在一個節點上運行多個映射器
雖然使用谷歌搜索，我發現通過使用conf.setNumReduceTasks(2);我可以改變減速器的數量。但我沒有看到我的輸出有任何改變。 （我的輸出是以毫秒爲單位的時間）。
我使用的代碼是來自github：https://github.com/himank/K-Means/blob/master/src/KMeans.java 雖然我根據我的要求做了一些更改，但主要功能是相同的。
這裏是如何的功能主要是這樣的：
public static void main(String[] args) throws Exception {
long startTime = System.currentTimeMillis();
IN = args[0];
OUT = args[1];
String input = IN;
String output = OUT + System.nanoTime();
String again_input = output;
int iteration = 0;
boolean isdone = false;
while (isdone == false) {
JobConf conf = new JobConf(KMeans.class);
if (iteration == 0) {
Path hdfsPath = new Path(input + CENTROID_FILE_NAME);
DistributedCache.addCacheFile(hdfsPath.toUri(), conf);
} else {
Path hdfsPath = new Path(again_input + OUTPUT_FILE_NAME);
DistributedCache.addCacheFile(hdfsPath.toUri(), conf);
}
conf.setJobName(JOB_NAME);
//conf.setNumReduceTasks(2);
conf.setMapOutputKeyClass(DoubleWritable.class);
conf.setMapOutputValueClass(DoubleWritable.class);
conf.setOutputKeyClass(DoubleWritable.class);
conf.setOutputValueClass(Text.class);
conf.setMapperClass(Map.class);
conf.setNumMapTasks(4);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(input + DATA_FILE_NAME));
FileOutputFormat.setOutputPath(conf, new Path(output));
JobClient.runJob(conf);
Path ofile = new Path(output + OUTPUT_FILE_NAME);
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://127.0.0.1:9000"), configuration);
Path filePath = new Path(output + OUTPUT_FILE_NAME);
BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(filePath)));
List<Double> centers_next = new ArrayList<Double>();
String line = br.readLine();
while (line != null) {
String[] sp = line.split("\t| ");
double c = Double.parseDouble(sp[0]);
centers_next.add(c);
line = br.readLine();
}
br.close();
String prev;
if (iteration == 0) {
prev = input + CENTROID_FILE_NAME;
} else {
prev = again_input + OUTPUT_FILE_NAME;
}
Path prevfile = new Path(prev);
FileSystem fs1 = FileSystem.get(new URI("hdfs://127.0.0.1:9000"), configuration);
BufferedReader br1 = new BufferedReader(new InputStreamReader(fs1.open(prevfile)));
List<Double> centers_prev = new ArrayList<Double>();
String l = br1.readLine();
while (l != null) {
String[] sp1 = l.split(SPLITTER);
double d = Double.parseDouble(sp1[0]);
centers_prev.add(d);
l = br1.readLine();
}
br1.close();
Collections.sort(centers_next);
Collections.sort(centers_prev);
Iterator<Double> it = centers_prev.iterator();
for (double d : centers_next) {
double temp = it.next();
if (Math.abs(temp - d) <= 0.1) {
isdone = true;
} else {
isdone = false;
break;
}
}
++iteration;
again_input = output;
output = OUT + System.nanoTime();
}
long endTime = System.currentTimeMillis();
long totalTime = endTime - startTime;
System.out.println(totalTime);
}
PS。我是Hadoop和MapReduce的新手。
從HDFS上讀取文件之前，您可能應該等待作業完成 –
@Ahsan：您是否在尋找性能調整的一部分，這就是爲什麼您想嘗試設置映射器和縮減器的數量 –
@ ramprasad-g是的，因爲我說我正在計算加速。我通過增加節點數來加速。現在我想通過增加mappers和reducer的數量來在單個節點上計算它。 –