如何提高此代碼的速度？

Question

我試圖將所有googlebooks-1gram文件導入到postgresql數據庫中。我寫了下面的Java代碼爲：

public class ToPostgres { 

    public static void main(String[] args) throws Exception { 
     String filePath = "./"; 
     List<String> files = new ArrayList<String>(); 
     for (int i =0; i < 10; i++) { 
      files.add(filePath+"googlebooks-eng-all-1gram-20090715-"+i+".csv"); 
     } 
     Connection c = null; 
     try { 
      c = DriverManager.getConnection("jdbc:postgresql://localhost/googlebooks", 
        "postgres", "xxxxxx"); 
     } catch (SQLException e) { 
      e.printStackTrace(); 
     } 

     if (c != null) { 
      try { 
       PreparedStatement wordInsert = c.prepareStatement(
        "INSERT INTO words (word) VALUES (?)", Statement.RETURN_GENERATED_KEYS 
       ); 
       PreparedStatement countInsert = c.prepareStatement(
        "INSERT INTO wordcounts (word_id, \"year\", total_count, total_pages, total_books) " + 
        "VALUES (?,?,?,?,?)" 
       ); 
       String lastWord = ""; 
       Long lastId = -1L; 
       for (String filename: files) { 
        BufferedReader input = new BufferedReader(new FileReader(new File(filename))); 
        String line = ""; 
        while ((line = input.readLine()) != null) { 
         String[] data = line.split("\t"); 
         Long id = -1L; 
         if (lastWord.equals(data[0])) { 
          id = lastId; 
         } else { 
          wordInsert.setString(1, data[0]); 
          wordInsert.executeUpdate(); 
          ResultSet resultSet = wordInsert.getGeneratedKeys(); 
          if (resultSet != null && resultSet.next()) 
          { 
           id = resultSet.getLong(1); 
          } 
         } 
         countInsert.setLong(1, id); 
         countInsert.setInt(2, Integer.parseInt(data[1])); 
         countInsert.setInt(3, Integer.parseInt(data[2])); 
         countInsert.setInt(4, Integer.parseInt(data[3])); 
         countInsert.setInt(5, Integer.parseInt(data[4])); 
         countInsert.executeUpdate(); 
         lastWord = data[0]; 
         lastId = id; 
        } 
       } 
      } catch (SQLException e) { 
       e.printStackTrace(); 
      } 
     } 
    } 

} 

然而，在運行此爲約3小時，此時它只能放在wordcounts表1.000.000條目。當我檢查整個1gram數據集中的行數時，它是500.000.000行。因此，導入所有東西需要大約62.5天，我可以接受它在大約一週內進口，但是2個月？我認爲我在這裏做了嚴重錯誤的事情（我有一臺運行24/7的服務器，所以我可以真正運行它很長時間，但更快會是很好的XD）

編輯：此代碼是我解決它：

public class ToPostgres { 

    public static void main(String[] args) throws Exception { 
     String filePath = "./"; 
     List<String> files = new ArrayList<String>(); 
     for (int i =0; i < 10; i++) { 
      files.add(filePath+"googlebooks-eng-all-1gram-20090715-"+i+".csv"); 
     } 
     Connection c = null; 
     try { 
      c = DriverManager.getConnection("jdbc:postgresql://localhost/googlebooks", 
        "postgres", "xxxxxx"); 
     } catch (SQLException e) { 
      e.printStackTrace(); 
     } 

     if (c != null) { 
      c.setAutoCommit(false); 
      try { 
       PreparedStatement wordInsert = c.prepareStatement(
        "INSERT INTO words (id, word) VALUES (?,?)" 
       ); 
       PreparedStatement countInsert = c.prepareStatement(
        "INSERT INTO wordcounts (word_id, \"year\", total_count, total_pages, total_books) " + 
        "VALUES (?,?,?,?,?)" 
       ); 
       String lastWord = ""; 
       Long id = 0L; 
       for (String filename: files) { 
        BufferedReader input = new BufferedReader(new FileReader(new File(filename))); 
        String line = ""; 
        int i = 0; 
        while ((line = input.readLine()) != null) { 
         String[] data = line.split("\t"); 
         if (!lastWord.equals(data[0])) { 
          id++; 
          wordInsert.setLong(1, id); 
          wordInsert.setString(2, data[0]); 
          wordInsert.executeUpdate(); 
         } 
         countInsert.setLong(1, id); 
         countInsert.setInt(2, Integer.parseInt(data[1])); 
         countInsert.setInt(3, Integer.parseInt(data[2])); 
         countInsert.setInt(4, Integer.parseInt(data[3])); 
         countInsert.setInt(5, Integer.parseInt(data[4])); 
         countInsert.executeUpdate(); 
         lastWord = data[0]; 
         if (i % 10000 == 0) { 
          c.commit(); 
         } 
         if (i % 100000 == 0) { 
          System.out.println(i+" mark file "+filename); 
         } 
         i++; 
        } 
        c.commit(); 
       } 
      } catch (SQLException e) { 
       e.printStackTrace(); 
      } 
     } 
    } 

} 

我現在約15分鐘達到150萬行。這對我來說足夠快，謝謝大家！

Answer 1

默認情況下，JDBC連接具有啓用自動提交的功能，該自動提交包含每個語句的開銷。嘗試禁用它：

c.setAutoCommit(false) 

然後提交分批，分東西線沿線的：

long ops = 0; 

for(String filename : files) { 
    // ... 
    while ((line = input.readLine()) != null) { 
     // insert some stuff... 

     ops ++; 

     if(ops % 1000 == 0) { 
      c.commit(); 
     } 
    } 
} 

c.commit(); 

Answer 2

編寫它來執行線程，同時運行4個線程，或者將它分成幾部分（從配置文件讀取）並將其分發到X機器，並讓它們獲取數據togeather。

Answer 3

使用batch statements在一個時間在同一時間執行多個刀片，而不是一個INSERT。

此外，我會刪除算法中每次插入到words表後更新字數的部分，而只是在插入words完成後計算所有字計數。

Answer 4

另一種方法是執行批量插入而不是單個插入。有關更多信息，請參閱此問題Whats the fastest way to do a bulk insert into Postgres?。

Answer 5

如果您的表有索引，刪除它們可能會更快，插入數據並在稍後重新創建索引。

設置自動提交關閉，並且每10 000條記錄進行一次手動提交（查看合理值的文檔 - 有一定限制）也可以加速。

自己生成索引/外鍵，並跟蹤它應該快於wordInsert.getGeneratedKeys();，但我不確定，從您的內容是否可能。

有一種稱爲「批量插入」的方法。我不記得細節，但它是搜索的起點。

Answer 6

創建線程

String lastWord = ""; 
    Long lastId = -1L; 
    PreparedStatement wordInsert; 
    PreparedStatement countInsert ; 
    public class ToPostgres { 
     public void main(String[] args) throws Exception { 
      String filePath = "./"; 
      List<String> files = new ArrayList<String>(); 
      for (int i =0; i < 10; i++) { 
       files.add(filePath+"googlebooks-eng-all-1gram-20090715-"+i+".csv"); 
      } 
      Connection c = null; 
      try { 
       c = DriverManager.getConnection("jdbc:postgresql://localhost/googlebooks", 
         "postgres", "xxxxxx"); 
      } catch (SQLException e) { 
       e.printStackTrace(); 
      } 

      if (c != null) { 
       try { 
        wordInsert = c.prepareStatement(
         "INSERT INTO words (word) VALUES (?)", Statement.RETURN_GENERATED_KEYS 
        ); 
        countInsert = c.prepareStatement(
         "INSERT INTO wordcounts (word_id, \"year\", total_count, total_pages, total_books) " + 
         "VALUES (?,?,?,?,?)" 
        ); 
        for (String filename: files) { 
         new MyThread(filename). start(); 
        } 
       } catch (SQLException e) { 
        e.printStackTrace(); 
       } 
      } 
     } 

    } 
    class MyThread extends Thread{ 
     String file; 
     public MyThread(String file) { 
      this.file = file; 
     } 
     @Override 
     public void run() {   
      try { 
       super.run(); 
       BufferedReader input = new BufferedReader(new FileReader(new File(file))); 
       String line = ""; 
       while ((line = input.readLine()) != null) { 
        String[] data = line.split("\t"); 
        Long id = -1L; 
        if (lastWord.equals(data[0])) { 
         id = lastId; 
        } else { 
         wordInsert.setString(1, data[0]); 
         wordInsert.executeUpdate(); 
         ResultSet resultSet = wordInsert.getGeneratedKeys(); 
         if (resultSet != null && resultSet.next()) 
         { 
          id = resultSet.getLong(1); 
         } 
        } 
        countInsert.setLong(1, id); 
        countInsert.setInt(2, Integer.parseInt(data[1])); 
        countInsert.setInt(3, Integer.parseInt(data[2])); 
        countInsert.setInt(4, Integer.parseInt(data[3])); 
        countInsert.setInt(5, Integer.parseInt(data[4])); 
        countInsert.executeUpdate(); 
        lastWord = data[0]; 
        lastId = id; 
       } 
      } catch (NumberFormatException e) { 
       e.printStackTrace(); 
      } catch (FileNotFoundException e) { 
       e.printStackTrace(); 
      } catch (IOException e) { 
       e.printStackTrace(); 
      } catch (SQLException e) { 
       e.printStackTrace(); 
      } 
     }