handle sql exception for large data insert

user_1518668 0
Viewed 78

I have a Spring 2.5 application that takes a large (275K) file and parses it. Each record is then inserted into a Postgres db. There is a unique column (not the primaryKey/@Id) that will kick out the attempted record insert. This results in a DataContraintViolationException, which seems natural enough.

The problem I have is this kills the process. Is there a good way to continue processing the entire file, and just log the exception and move onto the next record for insert? I tried wrapping the respository.save(record) in a try/catch, but it still kills the process with a transaction rollback.

spring-data-jpa
1 Answers

A ConstraintViolationException will be wrapped in a PersistenceException and Hibernate will generally mark the transaction for rollback - even if the exception was registered to not cause a rollback at the spring transaction handling level, e.g. via @Transactional(noRollbackFor = PersistenceException.class).

So there needs to be a different solution. Some ideas:

  1. explicitly look whether a corresponding row is already present (one additional select per item)
  2. try every insert in a dedicated transaction (e.g. annotating a corresponding service method with @Transactional(propagation = Propagation.REQUIRES_NEW) (one additional transaction per item)
  3. handle the constraint violation in a custom DB statement (e.g. ON CONFLICT DO NOTHING / other "upsert" / "merge" behavior the DB offers)

The 1st and the 2nd option should offer some potential for parallelization, since selects / inserts can be issued independently from each other and there is no need to wait for unrelated DB roundtrips.

The 3rd option could be the fastest, as it requires no selects, the least amount of DB roundtrips, and statements could be batched; however it probably also needs the most amount of custom setup: Spring JPA bulk upserts is slow (1,000 entities took 20 seconds) (Reporting back which number or even which entities were actually inserted would likely even increase the complexity: How can I get the INSERTED and UPDATED rows for an UPSERT operation in postgres)

user_131561260
Related