I run an app on AWS elastic beanstalk with php, EC2, and mysql 5.6 (RDS). There are currently about 5k hits/min each resulting in multiple db operations per hit. I have noticed that when ever I make a change in code or db and resume the service, the average latency starts very low (10ms range), but after few hours, i see a sudden jump to ~200mS. I am attaching some screen shots of the latency graph.
Given the codebase is running for hours with 10mS latency, I am confused as to whether it is the code itself causing this issue as it seems fine when starting off. I have been optimizing my code over time to fine tune resource usage, db calls, etc and it definitely has helped but I cannot figure out why this sharp jump in latency which occurs in few hours.
Even when latency goes up, the connections or cpu usage does not really move much.
Where can I start/ How can I go about troubleshooting this?
Update:
Variable_name Value
innodb_stats_auto_recalc ON
innodb_stats_include_delete_marked OFF
innodb_stats_method nulls_equal
innodb_stats_on_metadata OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages 20
innodb_stats_sample_pages 8
innodb_stats_transient_sample_pages 8