Keeping data from db syned with horizontal scale

user_1245668 0
Viewed 328

Lets assume we have a microservice "A". We are now scaling it horizontally, meaning we have 3 instances of "A" working on the same db instance (and schemea, in general assume the 3 "A" instances might do Read and Writes on the same data).

Now I'll demonstrate the question with some pseudo code, we have the following update function in "A":

Product p = getProdFromDb(); // for example selecting 
// from Postgresql db

p.updateInnerData(); // synch method that updates 
// something inside the p model that takes significant 
// amount of time
p.updateInDb(); //  for example update back in postgresql

The problem here is that other "A" instances might change product p while we are updating it here (not in this function, but asdume there such other functions that change products in "A"). One solution that I know of is using a lock on the db (for example using "Select ... for Update") but it creates a performance bottleneck in this function. I would like to see better solutions that solves this problem without this bottleneck, real examples in Java (or JS) would be very helpful.

Edit: assume partioning is not an option

horizontal-scaling microservices relational-database architecture java
3 Answers

There are two kinds of locking: pessimistic (the one that you try to avoid) and optimistic locking.

In optimistic locking you do not hold any lock but try to save the document; if the document was already modified at the same time (meaning it was modified since we loaded it) then you retry the entire process (load + mutate + save).

One way of doing it is to have a version column that is increased every time you mutate the entity. When you try to persist, you expect that the entity with version = version + 1 does not exists. If it already exists, then it means that a concurrent update has happened and you retry (load + mutate + save).

In pseudocode, the algorithm is like this:

function updateEntity(ID, load, mutate, create)

    do
    {
        entity, version = load(ID) or create entity
        entity = mutate entity
        updateRow(that matches the ID and version) and increment version
    }
    while (row has not changed and was not inserted)

I will give you also a code sample in PHP (hope it's easy to understand) for MongoDB:

class OptimisticMongoDocumentUpdater
{

    public function addOrUpdate(Collection $collection, $id, callable $hidrator, callable $factory = null, callable $updater, callable $serializer)
    {
        /**
         * We try to add/update the entity in a concurrent safe manner
         * using optimistic locking: we always try to update the existing version;
         * if another concurrent write has finished before us in the mean time
         * then retry the *whole* updating process
         */

        do {
            $document = $collection->findOne([
                '_id' => new ObjectID($id),
            ]);

            if ($document) {
                $version = $document['version'];
                $entity = \call_user_func($hidrator, $document);
            } else {
                if (!$factory) {
                    return;//do not create if factory does not exist
                }
                $entity = $factory();
                $version = 0;
            }

            $entity = $updater($entity);

            $serialized = $serializer($entity);

            unset($serialized['version']);

            try {
                $result = $collection->updateOne(
                    [
                        '_id'     => new ObjectID($id),
                        'version' => $version,
                    ],
                    [
                        '$set' => $serialized,
                        '$inc' => ['version' => 1],
                    ],
                    [
                        'upsert' => true,
                    ]
                );
            } catch (\MongoDB\Driver\Exception\WriteException $writeException) {
                $result = $writeException->getWriteResult();
            }

        } while (0 == $result->getMatchedCount() && 0 == $result->getUpsertedCount());//no side effect? then concurrent update -> retry
    }
}
user_25752240

In my answer, I assume you want 100% reliability.

If that's the case, you can divide your tables to many pages, where each page will contains X amount of rows. when you try to update a table, you will only lock that page, but then there will be more I/O.

Also, on your db, you can configure it so that a select command will read even uncommited rows, which will improve speed - for SQL server, it's SELECT WITH (NOLOCK)

user_102466660

If p.updateInnerData(); takes time X and you have a high IO rate that might send Y no new requests to the microservice, then the function itself is posing performance bottleneck. Ideally microservice db related operations should be designed crisp performance benchmarks in mind; this often leads us to choice of the database itself which enables the high/expected IOPS you want to achieve.

  1. With RDBMS being target database, one possible option might be to decouple the expensive db operation pertaining p.updateInnerData(); and make it async over a reliable messaging platform that ensures strict ordering w/o compromising speed; e.g. Kafka; we can even think of backing up the messaging through storing the p object/changes in itself like a BLOB/JSON in a Db table and immediately return the control to user and then triggering the message asynchronously.

  2. With NoSQL being target database, we would want to choose based on our READ/WRITE needs and thus largely reducing write and read related latency.

user_23717150
Related