將數據寫入每個數據節點中的本地磁盤

Question

我想將map任務中的某些值存儲到每個數據節點中的本地磁盤中。例如，

public void map (...) { 
    //Process 
    List<Object> cache = new ArrayList<Object>(); 
    //Add value to cache 
    //Serialize cache to local file in this data node 
} 

我怎麼能存儲此緩存對象中的每個數據節點的本地磁盤，因爲如果我的map函數存儲這個緩存像上面，那麼性能將是可怕的，因爲I/O任務？

我的意思是有什麼辦法可以等待這個數據節點中的map任務完全運行，然後我們將這個緩存存儲到本地磁盤？或者Hadoop是否有解決此問題的功能？

Answer 1

請看下面的例子，創建的文件將在NodeManager使用的容器的目錄下的某個地方。這是配置屬性yarn.nodemanager.local - 迪爾斯紗線-site.xml中，或者從紗default.xml中繼承的默認，這是下/tmp

Please see @Chris Nauroth answer, Which says that Its just for debugging purpose and It's not recommended as a permanent production configuration. It was clearly described why it was not recommended.

public void map(Object key, Text value, Context context) 
     throws IOException, InterruptedException { 
    // do some hadoop stuff, like counting words 
    String path = "newFile.txt"; 
    try { 
     File f = new File(path); 
     f.createNewFile(); 
    } catch (IOException e) { 
     System.out.println("Message easy to look up in the logs."); 
     System.err.println("Error easy to look up in the logs."); 
     e.printStackTrace(); 
     throw e; 
    } 
}