Read large mongodb data

user_4561230 0
Viewed 4398

I have a java application that needs to read a large amount of data from MongoDB 3.2 and transfer it to Hadoop.

This batch application is run every 4 hours 6 times a day.

Data Specifications:

  • Documents: 80000 at a time (every 4 hours)
  • Size : 3gb

Currently I am using MongoTemplate and Morphia in order to access MongoDB. However I get an OOM exception when processing this data using the following :

List<MYClass> datalist = datasource.getCollection("mycollection").find().asList();

What is the best way to read this data and populate to Hadoop?

  • MongoTemplate::Stream() and write to Hadoop one by one?
  • batchSize(someLimit) and write the entire batch to Hadoop?
  • Cursor.batch() and write to hdfs one by one?
morphia mongodb hadoop java
2 Answers

Your problem lies at the asList() call

This forces the driver to iterate through the entire cursor (80,000 docs few Gigs), keeping all in memory.

batchSize(someLimit) and Cursor.batch() won't help here as you traverse the whole cursor, no matter what batch size is.

Instead you can:

1) Iterate the cursor: List<MYClass> datalist = datasource.getCollection("mycollection").find()

2) Read documents one at a time and feed the documents into a buffer (let's say a list)

3) For every 1000 documents (say) call Hadoop API, clear the buffer, then start again.

user_19135370

The asList() call will try to load the whole Mongodb collection into memory. Trying to make an in memory list object bigger than 3gb of size.

Iterating the collection with a cursor will fix this problem. You can do this with the Datasource class, but I prefer the type safe abstractions that Morphia offers with the DAO classes:

  class Dao extends BasicDAO<Order, String> {
    Dao(Datastore ds) {
      super(Order.class, ds);
    }
  }

  Datastore ds = morphia.createDatastore(mongoClient, DB_NAME);
  Dao dao = new Dao(ds);

  Iterator<> iterator = dao.find().fetch();
  while (iterator.hasNext()) {
      Order order = iterator.next;
      hadoopStrategy.add(order);
  }
user_76652520
Related