I am facing sudden freezes in one of my instances.
Here are some details about the instance:
Type: t2.micro
Region: ap-southeast-1
Platform: Ubuntu
Platform details: Linux/UNIX
I have two apps running on this instance, and also I have a celery service working there. these freezes start to happen since I run the celery service.
I am not really sure how to find and trace the issue.
Thanks.
When an EC2 instance freezes that is quite often due to it running out of memory. So what you can do is to enable memory metrics (there is a guide in the AWS documentation how to do it) on your EC2 Celery nodes, and next time this happens check memory utilisation to confirm that machine was out of memory when it froze.
You instance probably has high memory usage (probably memory leak) or high CPU usage (we had also this case)
you can use the tool 'atop' to monitor your CPU, Memory resources. This tool gives historical data of which process was utilizing maximum resources For more information, please refer the below links :
[+] https://www.tecmint.com/how-to-install-atop-to-monitor-logging-activity-of-linux-system-processes/ [+] https://haydenjames.io/use-atop-linux-server-performance-analysis/
You have mentioned that the issue gets resolved on rebooting the instance. It might be possible that memory crunch is happening on the instance. Rebooting the instance frees up the resources for sometime. However, as soon as the processes start coming up they may start consuming more resources making the instance unresponsive again and again.
I would recommend you to install(if already not installed) tools like atop, which will store the historical data of resource utilization and that will helps to find out more details like which process was consuming most of the resource etc and it will helps to troubleshoot the issue better. If based on atop logs you believe that your application requires more resources than offered by t2.micro then you may consider upgrading the instance type to one which offers more cpu/memory