春季數據JPA批量插入非常慢

Question

我想用700K +記錄讀取Excel文件並批量插入MySQL數據庫表中的那些文件。

請注意，Excel解析速度很快，我可以在50秒左右的時間內將我的實體對象放入ArrayList。

我正在使用Spring Boot和Spring Data JPA。

下面是我的部分application.properties文件：

hibernate.jdbc.batch_size=1000 
spring.jpa.hibernate.use-new-id-generator-mappings=true 

和我的部分Entity class：

@Entity 
@Table(name = "WHT_APPS", schema = "TEST") 
public class WHTApps { 

    @Id 
    @TableGenerator(name = "whtAppsGen", table = "ID_GEN", pkColumnName = "GEN_KEY", valueColumnName = "GEN_VAL") 
    @GeneratedValue(strategy = GenerationType.TABLE, generator = "whtAppsGen") 
    private Long id; 

    @Column(name = "VENDOR_CODE") 
    private int vendorCode; 
    . 
    . 
    . 
    . 

下面是我DAO：

@Repository 
@Transactional 
public class JapanWHTDaoImpl implements JapanWHTDao { 

    @Autowired 
    JapanWHTAppsRepository appsRepo; 

    @Override 
    public void storeApps(List<WHTApps> whtAppsList) { 
     appsRepo.save(whtAppsList); 

    } 

及以下Repository類：

@Transactional 
public interface JapanWHTAppsRepository extends JpaRepository<WHTApps, Long> { 

} 

有人請賜教我，我在做什麼不正確嗎？

編輯：

過程沒有完成，最終拋出錯誤： -

2017-08-15 15:15:24.516 WARN 14710 --- [tp1413491716-17] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08S01 
2017-08-15 15:15:24.516 ERROR 14710 --- [tp1413491716-17] o.h.engine.jdbc.spi.SqlExceptionHelper : Communications link failure 

The last packet successfully received from the server was 107,472 milliseconds ago. The last packet sent successfully to the server was 107,472 milliseconds ago. 
2017-08-15 15:15:24.518 INFO 14710 --- [tp1413491716-17] o.h.e.j.b.internal.AbstractBatchImpl  : HHH000010: On release of batch it still contained JDBC statements 
2017-08-15 15:15:24.525 WARN 14710 --- [tp1413491716-17] c.m.v.c3p0.impl.DefaultConnectionTester : SQL State '08007' of Exception tested by statusOnException() implies that the database is invalid, and the pool should refill itself with fresh Connections. 

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown. 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_131] 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_131] 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_131] 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_131] 
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.43.jar:5.1.43] 
    . 
    . 
    . 
    . 
    2017-08-15 15:15:24.526 WARN 14710 --- [tp1413491716-17] c.m.v2.c3p0.impl.NewPooledConnection  : [c3p0] A PooledConnection that has already signalled a Connection error is still in use! 
2017-08-15 15:15:24.527 WARN 14710 --- [tp1413491716-17] c.m.v2.c3p0.impl.NewPooledConnection  : [c3p0] Another error has occurred [ com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown. ] which will not be reported to listeners! 

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown. 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_131] 

感謝

Answer 1

我想指出一兩件事。問題可能不僅僅是休眠而是數據庫。

當您在一個事務中插入700k個對象時，它可以存儲在數據庫的回滾段中，等待事務提交。

如果可能的話，將邏輯拆分爲在中間提交。

從主列表創建1k大小的子列表，保存子列表並在每個子列表保存後提交。