retry buffer in rxjava

user_356083 0
Viewed 137

A hot Observable emits items. I want to upload these items to the server. There are two considerations:

  1. Due to expense of io operations, I want to batch these items and upload as array
  2. Due to unreliability of io operations, I want failed uploads of batches to be prepended to next batch.
Uploads succeed:
1 - 2 - 3 - 4 - 5
------------------
u(1,2,3) - u(4,5)

First upload fails:
1 - 2 - 3 - 4 - 5
------------------
u(1,2,3) - u(1,2,3,4,5)

I can take care of the first by using the buffer operator, but don't know how to satisfy the second requirement.

rx-java2 rx-java buffering
1 Answers

Here is my idea of storing failures in a queue

public class StackOverflow {

    public static void main(String[] args) {
        // store any failures that may have occurred
        LinkedBlockingQueue<String> failures = new LinkedBlockingQueue<>();

        toUpload()
                // buffer however you want
                .buffer(5)
                // here is the interesting part
                .flatMap(strings -> {
                    // add any previous failures
                    List<String> prevFailures = new ArrayList<>();
                    failures.drainTo(prevFailures);
                    strings.addAll(prevFailures);

                    return Flowable.just(strings);
                })
                .flatMapCompletable(strings -> {
                    // upload the data
                    return upload(strings).doOnError(throwable -> {
                        // if its an upload failure:
                        failures.addAll(strings);
                    });
                }).subscribe();
    }

    // whatever your source flowable is
    private static Flowable<String> toUpload() {
        return Flowable.fromIterable(Arrays.asList("a", "b", "c", "d", "e", "f", "g", "h", "i"));
    }

    // some upload operation
    private static Completable upload(List<String> strings) {
        return Completable.complete();
    }
}

some edge cases here are teh fact that if the last flowable buffered group fails, this won't retry those. That could maybe be achieved through a retryWhen operator but the basic idea applies the same of using the queue

user_15728480
Related