Spring data timeout based batch insert of data

devesh joshi 0
Viewed 19

I have a spring boot app that receives data from different devices that I need to insert as batch to maximize performance. For this I have considered storing data in memory for a while and then calling build in spring data saveAll method to save as a batch. But the my problem is that data does not come at a regular speed and I may get a few thousand data at a time and then no data for next few hours. So I am not sure how to properly build the batch as there may be few data at the end that would be left out with a fixed sized batch. Is there a way to do a timed batching in java/spring that flushes the batch after a configured timeout?

batch-processing spring-data-jpa java
1 Answers

I am currently doing this using BlockingQueue and Guauva's Queues.drain

executorService.submit(() -> {
            while (true) {
                try {
                    String first = queue.take();
                    List<String> output = new LinkedList<>();
                    output.add(first);
                    Queues.drain(queue, output, BATCH_SIZE - 1, 5, TimeUnit.SECONDS);
                    repo.saveAll(output);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });

Where I am doing queue.add() from the producer thread

devesh joshi0
Related