What is the best way to update the large data set on single request

user_5219790 0
Viewed 97

I am working on Social networking application with at least 0.2M users. On the application user can share anything from third parties as well as user can upload own media as post. There are different types of privacy

  • user privacy
    • user can be public
    • user can be private Any content shared or uploaded by the user will be in a box, and box also has different types of privacy
  • public box (Everyone can see the content of this box if you are public)
  • friend only box (Only your followers can see the content of this box)
  • private (Only you can see the content of this box)

Now the problem is that I have large data set. So when a user change his/her account privacy from public to private or private to public I have to update all the data according to privacy. Also, user can change the privacy of the box too. So I need to update the user all shared posts of this box accordingly. But most of the time I failed to update due to framework and also technologies that I am using Technologies that I am using

  • Lumen (PHP) microservice architecture
  • MySQL
  • Elasticsearch (For retrieval with joins)
  • Redis & Memcached
  • Postgres

When user shares anything on the platform the shared data is stored in the database and also data inserted in elasticsearch so all the data retrieved from elasticsearch with PHP client.

Now I want to define the architecture like Instagram that whenever user change account privacy or box privacy I have to change the content according to both privacies.

I read different types of articles but didn't get any close idea for this. Kindly suggest any helpful article or idea to me.

microservices elasticsearch laravel backend architecture
1 Answers

I would agree with @KolovosKonstantinos, you should try to model your application data to avoid updates of large data sets. Also it might be interesting for you to check our concept of Embedded vs Referenced Documents. Here are couple of nice posts on this topic :

I would suggest trying following approach :

  • User entity has privacy property and different collections of post ids for every type of box
  • Every post stored as a separate document in elastic
  • You have different queries to select post ids from different collections based on the user privacy settings. When privacy of post changed simply move its id from one collection to another. Yes you need second query (sometimes called roundtrip) to storage to retrieve post when you know its id. That's a tradeoff you make. Defining architecture is all about tradeoff.

So you data might look like as following :

User document:
{
    "userId": "1",
    "privacy": "Public",
    "publicBoxPostIds":  [1,3],
    "friendsBoxPostIds": [2],
    "privateBoxPostIds": [],
}

Post documents:
{
    "postId": "1",
    "postText": "zbzb",
     ...
},
{
    "postId": "2",
    "postText": "xcxc",
     ...
}
...
user_196000630
Related