Abusing Git for implementation of Event Store architecture?

user_11389375 0
Viewed 383

Git is essentially an implementation of an Event Store where the data stored are files in a directory structure. It is known to reliably solve the problems:

  • Store history of changes
  • Transfer minimal data to client to get most recent data
  • Can rollback to previous state

It is possible to create an Event Store by writing a wrapper over Git.

Suppose my business needs are that I need to store customer data that can be represented in JSON format. The data can be modified by one or more services in the system. I could have a dedicated Git repo Customer-Data with flat structure and files named {customer-id}.json. When a service modifies the data it includes a useful commit message.

This solution does not scale (if there are too many customers with too frequent of changes, the remote Git service, e.g. GitHub, will get bombarded with requests and throttle), but assuming I know that I will have ~1,000 customers and data changes of 1 per ~10 hours per customer, are there any other problems with the solution?

event-sourcing architecture events github git
2 Answers

Using Git as a database is generally a bad idea. It isn't especially optimized for this use case, since it writes more data than is typically required for a database transaction, generally wants the entire tree to be checked out, and is difficult to shard if you need to scale in the future. It also can't be operated in a multiple-master mode for replication and scalability.

Furthermore, if you do this anyway, your history will grow large in a pathological way that makes packing and repacking extraordinarily expensive in terms of CPU and memory due to the way Git deltifies objects. At that point, your Git hosting provider will notice and ask you to move elsewhere, at which point you'll need to switch to a real database.

user_87054320

Git is essentially an implementation of an Event Store where the data stored are files in a directory structure.

Sort of -- the git repository gives you snapshots of the work-tree with happens-before relationships that allow you to track the lineage.

On its own, it's not particularly good at the semantics. See the discussions of task-based-ui if you need more context, but effectively you are left with an analog to "writing good commit messages" to describe the changes you are making to the snapshot representations.

It's also decentralized, by design -- which is potentially awkward when what you want is a central authority for capital-T Truth. With decentralized authority, you are necessarily more reliant on memories, guesses, and apologies. That's not bad, necessarily, but it could be a nasty surprise if you weren't already budgeting for it.

When there is a clear responsibility for individual documents in the work-tree, that may relieve some of the pressure -- assuming that the latency between changes in different parts of the tree is acceptable.

user_547340
Related