Mongo - Tree data with frequent full load

Question

I am using mongodb and storing tree data( MongoDB is the only option for now ).

      10     ->>> Root node
      /\
     /  \
    8    6  ---->> 8 & 6 child node of 10
    /\    /\
   /  \  /  \
  4    5 2   1  ---->> 4 & 5 child node of 8 ...

Each node is a separate document in mongoDB and each document has bunch of fields.

Sample data,

{
  "_id": "234463456453643563456",
  "name": "Mike",
  "empId": "10",
  "managerId": "8",
  "hierarchy": [
     8,
     10
   ]
  "projects" : [ "123", "456", "789"]
}

Here, hierarchies field will have manager ids from 1st level to top level.

Any document might get updated with any field and node might move to any location. Basically, an org change.

I have a use case where changes will be captured in other system and my system will be updated with the full active load( 200k records out of 800k records ) every 2 hours.

Here, if there is any org change like, 8 is moving under 6, the bottom to top hierarchy will change for all nodes under 8. If the full load failed in b/w the org hierarchy result will not be correct until the complete load is done.

The result should be either before the full update or after the full update not in b/w. I was thinking on versioning to handle this. Is there any better way to handle this with mongo?

There are about 200k records for full load. But, the actual changes might be less than 1k record many times which we dont know.

user_14457033 · Answer

If you need an all-or-nothing (atomic) database update, where your database clients must not read an invalid mid-update state, then you need a transaction.

You can optimize by recognizing that some subsets of the graph are valid after you update them, and so queries against that subset of the graph is valid, and then you don't need to use the transaction feature of the database.

But you'll still be blocking or rejecting queries from some clients, and that makes your schema, queries or architecture more complex.

If this is a business issue, then I'd push on the business requirements. (If you're in a position to do that, you didn't say whether that was an option.)

Your clients are already reading data that are potentially 2 hours out of date. If the batch update you're applying is sorted, then you can make those updates in time-order, and your clients will always be receiving a state that was recently valid (but maybe not the most recent).

Mongo - Tree data with frequent full load

1 Answers