Java使用Byteoutputstream寫入大文件

Question

我正在嘗試使用ByteArrayOutputStream寫入大小在1kb到10GB之間的文件，但會引發下面的異常。我正在使用jdk 6.請建議任何更好的高性能Api。我正在使用相同的網絡盒來讀取和寫入。

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit 
     at java.util.Arrays.copyOf(Unknown Source) 
     at java.io.ByteArrayOutputStream.grow(Unknown Source) 
     at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) 
     at java.io.ByteArrayOutputStream.write(Unknown Source) 
     at java.io.OutputStream.write(Unknown Source) 
     at 

代碼：

import java.io.BufferedOutputStream; 
import java.io.ByteArrayOutputStream; 
import java.io.File; 
import java.io.FileInputStream; 
import java.io.FileOutputStream; 


public class PrepareFile { 

    /** 
    * @param args 
    * @throws Exception 
    */ 
    public static void main(String[] args) throws Exception { 
     // TODO Auto-generated method stub 

     new PrepareFile().constructFile("f:\\hello","f:\\output",10000000); 

    } 

    //Writes a large file of 10 GB using input file data of small size by duplicating 
    public void constructFile(String fileName, String outPath, int multiplier) throws Exception { 
     BufferedOutputStream fos = null; 
     FileInputStream fis = null; 

     final File inputFile = new File(fileName); 
     String path = inputFile.getParent(); 
     if (outPath != null && !outPath.isEmpty()) { 
      path = outPath; 
     } 

     fis = new FileInputStream(fileName); 

     try { 



      // read the transactions in the input file. 
      byte[] txnData = new byte[(int) inputFile.length()]; 
      fis.read(txnData); 

      final File outFile = new File(path, "Myfile"); 
      fos = new BufferedOutputStream(new FileOutputStream(outFile)); 
      final ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
      final ByteArrayOutputStream baos1 = new ByteArrayOutputStream(); 

      //multiplier if input file size is 1 KB and output file is 10 GB, then multiplier value is (1024*1024) 

      for (long i = 1; i <= multiplier; i++) { 

       if(i >=40000 && i % 40000==0){ 
        System.out.println("i value now: "+i); 
        baos.writeTo(fos); 
        baos.reset(); 
        //baos.write(txnData); 
       } 

       // write transactions 
       baos.write(txnData); 
       baos1.write(txnData); //Exception is coming at this line 
      } 

      int Padding = myCustomMethod(baos1.toByteArray()); 

      // write all out data to the output stream 
      baos.writeTo(fos); 

      baos.flush(); 
      baos1.flush(); 
     } catch(Exception e){ 
      e.printStackTrace(); 
     }finally { 
      fos.close(); 
      fis.close(); 
     } 

    } 

    public int myCustomMethod(byte[] b){ 

     //Need complete bytes to prepare the file trailer 
     return 0; 
    } 


} 

Answer 1

你不能有一個ByteArrayOutputStream 2 GB或更大的緩衝區大小爲32位有符號。

如果你想要的性能，我會逐步處理文件，並避免如此大的內存拷貝，因爲它們非常昂貴。

順便說一句我有一個庫Chronicle Bytes它支持大於2 GB的緩衝區，並且可以使用本地內存並映射到文件以避免使用堆並且可能比主內存大。

但是，如果逐步處理數據，則不需要如此大的緩衝區。

我也建議你使用Java 8，因爲它的性能比Java 6中（這是發佈在十年前）

---

編輯基於您的代碼更好的64位操作，沒有必要使用ByteArrayOutputStream並且您可以逐步準備文件。

//Writes a large file of 10 GB using input file data of small size by duplicating 
public void constructFile(String fileName, String outFileName, int multiplier) throws IOException { 
    byte[] bytes; 
    try (FileInputStream fis = new FileInputStream(fileName)) { 
     bytes = new byte[fis.available()]; 
     fis.read(bytes); 
    } 

    try (FileOutputStream fos = new FileOutputStream(outFileName)) { 
     for (int i = 0; i < multiplier; i++) { 
      fos.write(bytes); 
     } 
    } 

    // now process the file "outFileName" 
    // how depends on what you are trying to do. 
    // NOTE: It is entirely possible the file should be processed as it is written. 
}