Avoid Overflowing Consumer in RxJava2

Viewed 12

I am using vertx and rxjava 2 to import a file to mongodb.

The relevant part is here:

retrieveProcessableContent()
.flatMapSingle { jsonObject ->
            lookupInMongoDB(jsonObject) // returns Single<JsonObject>
        }
        .subscribe(
            {
                logger.info("lookup succeeded")
            },
            {
                logger.error("Error", it)
            }
        )

The method retrieveProcessableContent returns an Observable like this:

fun retrieveProcessableContent() : Observable<JsonObject> {
  // here jsonArray contains approx. 20,000 JsonObjects
  Observable.fromIterable(jsonArray.map { it as JsonObject })
}

For each jsonObject I then call the method lookupInMongoDB which does a mongodb query. It then returns a Single.

My issue is that after some time I get lots of the following errors:

com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded.

The problem is, that the stream seems to be processed in parallel and too many lookups are done the same time.

Is there a way to throttle the stream somehow so that it stops processing e.g. when 100 lookups are in progress and waits till the lookup finished?

I already tried with Flowable instead of Observable but with the same result.

My understanding was, that this should create a cold observable and so the items are emitted one after the other?!

Best regards, Marco

0 Answers
Related