I'm working on an game application where I need real-time data for leaderboards I'm building. I've read a bunch of stackoverflows and company blogs- but honestly, I'm not sure which one best fits my use case. I am using DynamoDB to record players' recent moves, and the history of moves are in kafka. I am looking to stream data from these two data sources into a database and my leaderboard-service can then query the database to render the contents of each leaderboard. My data velocity is modest (1K game events/sec). I find these three different databases that I can use, has anybody used any of these database for game-leaderboarding? If so, can you share the advantages or pains that you have encountered while doing so? According to all 3 companies, they are able to do real-time data.
1 Answers
You would have to evaluate the scale and performance that you require and it is difficult for me to estimate those based on the data you provided. But I can do a feature comparison of using some of these systems.
The first option is to run your leaderboards by querying DynamoDB itself and you do not need any additional systems. The advantage obviously is that there is one less component for you to manage. But I am assuming that your leaderboards need complex logic to render, and because DynamoDB api deals with key/values, you have to fetch a lot of data from DynamoDB to execute every query to render the leaderboard.
The second option you specified is Elastic Search. Great system that gives query results really fast because it stores data as an inverted index. However, you wont be able to do JOINs between your DynamoDB data and kafka stream. But you sure can run a lot of concurrent queries on Elastic. I am assuming that you need concurrent queries because you are powering an online game where m multiple players are accessing the leaderboard at the same time.
The third option, Druid, is a hybrid between a datalake and a data warehouse. You can store large volumes of semi-structured data, but unlike Elastic, you would need to flatten the nested json data an ingest time. I have used Druid for large scale analytical processing for powering my dashboards, and it does not support as high a concurrency as Elastic.
Rockset seems to be much newer product and is a hosted service on the cloud. It says that it build inverted index like Elastic and also supports JOINs. It can auto-tail data from DynamoDB (using change-streams) and kafka. I do not see any performance numbers on the website, but the functionality is very compatible with what I would need for building a game leaderboard.
Related