Problems with Java Spring Pagination, some pages are empty

Viewed 348

I have a problem with Java pagination. In my code I update some records from my database (MongoDB), but I'm trying to use pagination because the process is taking too much time. So, basically I'm using something like this:

int npages = 10;
int count = (query for count records);
int lote = count / npages;

for (int i = 0; i < npages + 1; i++) {

  Pageable page = PageRequest.of(i, lote);

  List < Example > example;
  example = exampleRepository
    .findByExample(parameter, page);

  if (null != example) {
    log.info("For i: " + i + " size: " + example.size());
  }

  new Thread(() -> {

    try {
      log.info("Start thread");
      process(example);
    } catch (Exception e) {
      log.error("Error processing: " + e.getMessage());
    }
  }).start();

}

So, I decided to have 10 pages, so I make a count of all the records (that query is OK, I checked the database and the result was the correct), then I use a for, I make the page there with it's size, and I do the query with Pageable page.

The problem is that sometimes, the last page and the previous one, are empty. For example, if I have 1235 records, I'm going to have 10 pages, 9 with 123 and the last one with 5. But sometimes I have 8 with 123, one with maybe 100 or something like that, and one with 0.

I'm remarking the sometimes thing for two reasons, I'm already using this pagination in another two process, working fine in both. And because, in this process, sometimes the pages are complete (and is not when I have something like 100/10 = 10. No, with inequal size in the last page works properly too, sometimes, but not always). So, I can't understand what's happening.

If I run the process again, it is going to take and process the missing records. I need that in the first execution.

For example, this is one of the executions, I logged the count and the lots size:

INFO  - [http-nio-8080-exec-3] - Count: 954 Lote: 95
INFO  - [http-nio-8080-exec-3] - For i: 0 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 1 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 2 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 3 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 4 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 5 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 6 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 7 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 8 sumAmount.size(): 95
INFO  - [http-nio-8080-exec-3] - For i: 9 sumAmount.size(): 5
INFO  - [http-nio-8080-exec-3] - For i: 10 sumAmount.size(): 0

So 95*9 = 855 + 5 = 860. 94 records missing. And I'm sure about this number. In this attempt I was using the test environment database, but I also tried with local data. With 52 records, some executions worked just fine, but I remember that I had one with 5 lots of 5, and the rest empty.

Queries:

Count:

@Aggregation(pipeline = { "{ $match : { 'example.date': { $lt: ?0}, 'example.state': {$elemMatch: {name: 'Aceptado' }}}  }", "{ $group: {_id : null, count : {$sum : 1}}}"})
    int count(String date);

Data:

List<RewardAccepted> findByExampleStateNameAndExampleDateLessThan(String exampleStateName ('Aceptado' of course), String date, Pageable pageable);

(date is the same for both)

1 Answers

SOLVED: I found the problem, it wasn't a pagination issue, when I started to process my data (after paging), I changed a field that was also a parameter for the query, so the collection started to decrease, and the position of the data to change, so the offset stopped making sense.

Related