How does real-time collaborative applications saves the data?

user_6943479 0
Viewed 121

I have previously done some very basic real-time applications using the help of sockets and have been reading more about it just for curiosity. One very interesting article I read was about Operational Transformation and I learned several new things. After reading it, I kept thinking of when or how this data is really saved to the database if I were to keep it. I have two assumptions/theories about what might be going on, but I'm not sure if they are correct and/or the best solutions to solve this issue. They are as follow:

(For this example lets assume it's a real-time collaborative whiteboard:)

  1. For every edit that happens (ex. drawing a line), the socket will send a message to everyone collaborating. But at the same time, I will store the data in my database. The problem I see with this solution is the amount of time I would need to access the database. For every line a user draws, I would be required to access the database to store it.
  2. Use polling. For this theory, I think of saving every data in temporal storage at the server, and then after 'x' amount of time, it will get all the data from the temporal storage and save them in the database. The issue for this theory is the possibility of a failure in the temporal storage (ex. electrical failure). If the temporal storage loses its data before it is saved in the database, then I would never be able to recover them again.

How do similar real-time collaborative applications like Google Doc, Slides, etc stores the data in their databases? Are they following one of the theories I mentioned or do they have a completely different way to store the data?

socket.io real-time canvas database
1 Answers

They prolly rely on logs of changes + latest document version + periodic snapshot (if they allow time traveling the document history).

  • It is similar to how most database's transaction system work. After validation the change is legit, the database writes the change in very fast data-structure on disk aka. the log that will only append the changed values. This log is replicated in-memory with a dedicated data-structure to speed up reads.

  • When a read comes in, the database will check the in-memory data-structure and merge the change with what is stored in the cache or on the disk.

  • Periodically, the changes that are present in memory and in the log, are merged with the data-structure on-disk.

So to summarize, in your case:

  • When an Operational Transformation comes to the server, two things happens:
  1. It is stored in the database as is, to avoid any loss (equivalent of the log)
  2. It updates an in-memory datastructure to be able to replay the change quickly in case an user request the latest version (equivalent of the memory datastructure)

  • When an user request the latest document, the server check the in-memory datastructre and replay the changes against the last stored consolidated document that might be lagging behind because of the following point

  • Periodically, the log is applied to the "last stored consolidated document" to reduce the amount of OT that must be replayed to produce the latest document.

Anyway, the best way to have a definitive answer is to look at open-source code that does what you are looking for, e.g. etherpad.

user_1408370
Related