EC2 goes down when db backup cron is running

user_6133422 0
Viewed 33

We are using T3 small instance of EC2 running ubuntu 18.04. It is having one local Mongo service, some node services. At 19:30 IST, the cron is scheduled to dump the Mongo database and store it in GCS cloud.

Sometimes this cron is causing the server to shut down, though not everytime.

I checked the cloudwatch metrics and read ops are around 10k+ at that time and burst balance also going down.

Is there any way to mitigate this?

devops amazon-web-services amazon-ec2
1 Answers

If your Read Ops are high, you would probably also see a large number of queued tasks at the same time which will cause performance to drop.

The most costly solution would be to upgrade your disk to PIOPs and set the number of IOPs as the highest consumption point, but this will have a cost impact on your solution.

The following steps will help to mitigate performance:

  • Do not run the backup on the mongodb server, run it from a standalone server.
  • Add a standby replica and backup from that as to not affect the performance of your primary node.
  • Add pauses between backup actions, rather than everything at once can you backup over a longer period with gaps in between to improve the overall performance.
user_134609330
Related