Background

I'm using Spring Batch to fetch data from our customer sites through HTTP API. The progress contains 2 main steps:

1. Fetch the total documents from API, then calculate the total pages using a configurable page size. Each page will be assigned to one partition step using custom Paritioner.
2. A partition step will send a request to fetch page of data (a list of documents), process and write to our storage.

Customer sites might be "fragile". They could have rate limit or their sites might not respond after some heavy requests.

What I have done so far

I'm using spring-retry to re-run a request which is failed because of rate limit or server error. For e.g:

// the partition step's item reader
@StepScope
public class CustomItemReader extends ItemReader<Object> {
  private List<Object> items;
  @Override
  public Object read() {
    if (Objects.isNull(items)) {
      this.items = ImportService.getPage(pageId);
    }
    if (Objects.nonNull(items) && !items.isEmpty()) {
      return items.remove(0);
    }
    return null;
  }
}

// config retry for fetching function
public class ImportService {
  @Retryable(
      value = RetryableException.class,
      maxAttempts = 3,
      backoff = @Backoff(
          delay = 1000
      )
  )
  public static List<Object> getPage(String pageId) throws RetryableException {
    return ...;
  }
}

The retry config contains Backoff policy, which has an incremental delay (1000 ms). I used this Retryable to handle both retry and rate limit.

Problem

1. Retryable will repeatedly wait and re-execute the function, which hold the thread for the whole time. The instance might crash when things get bigger.
2. Because each customer will have its own rate limit, using Retryable with Backoff is not an ideal way to control the rate. Eventhough I config core_pool_size for each customer sites, core_pool_size=1 is not enough for some.

Question

1. Is there any proper way to throttle the execution rate of Spring Batch, especially with Partitioning? For e.g: I want to config to send 2 requests in 10 seconds, and this will not be achieved by using sleep in step listener.
2. I have used scrapy for some crawlers, and it has pretty cool retry and rate limit features. With RetryMiddleware, it will enqueue the failed pages and has a RETRY_LIMIT in settings. With AutoThrottle, it can automatically throttle speed based on load on server. Is there any way to achieve kind of those features in Spring Batch? Or I have to rewrite my project with scrapy?

Thanh you very much!