MongoDB cursor is slow when fetching second batch

I'm seeing an odd behavior when iterating over a cursor: fetching the second batch is very slow (several seconds or more). The first batch is reasonably fast, as are all the batches after the second one. And the other odd thing is, I can make this behavior go away by increasing the batch size (default is 100, I increased it to 800).

    MongoCursor<Document> it = collection
            .find()
            .batchSize(800)
            .projection(Projections.fields(Projections.include("year")))
            .iterator();
    int count = 0;
    while (it.hasNext()) {
        System.out.println(""+count+" : "+it.next());
        count++;
    }

For the above example, I'm using a DB with about half a million records. If I don't set the batch size, it pauses after printing the 100th record, then continues normally.

Can someone explain why this is happening? Is it a bug in the server or the client? Does it indicate a problem with the way I set up my DB?

Server version: 4.0.10

Java client version: 3.10.1