I want to read all documents (events) from the large MongoDb collection (e.g. millions of documents) and process them in the same time (deserialize each event). This can be easily done with MongoDb C# driver but it is tricky to do it in the optimal way (in my case optimal means fastest way).
Through several iteration I came up with the solution which includes batch read, double buffering and parallelized deserialization of events.
As you can see in the code events are read in batch of 1000 and I use double buffered read here (in the same time new batch is read while the old batch is deserialized in parallel and returned through IEnumerable).
public IEnumerable<IReadOnlyList<IDomainEvent>> LoadAll()
{
using (var cursor = _mongoCollection.FindSync(Filter.Empty, FindOptionsFor1000BatchSize))
{
cursor.MoveNextAsync().Wait();
Task<bool> cursorTask;
do
{
var events = cursor.Current;
cursorTask = cursor.MoveNextAsync();
yield return events.AsParallel().Select(e => e.ToDomainEvent()).ToList();
} while (cursorTask.Result);
}
}
With this approach I was able to read and deserialize 1 000 000 events for 8 seconds. Detailed results:
- Batch read (1m events) - 15 sec
- Batch read with parallelization (1m events) - 11 sec
- Batch read with parallelization and double buffering (1m events) - 8 sec
I played with different batch sizes and in my case 1000 is the optimal size.
Is there any faster way of reading events from the MongoDb?