I'm having issue running load testing on my NodeJs and MySql application at 5k concurrency.
During my load testing all servers doesn't hit the limit but response time increase tremendously. Normally my api response around ~80ms, but when it hits 5k concurrent the response time increase to ~10s (Please note: this response is from logging in API console, NOT total response time received by client). At first I thought there was a problem connecting from API Servers to Database, so I set up Elastic Search APM to monitor the operations and see only it only spent around 0.8% with MySql and 99% on api instead. Below is my infrastructure and technology setup:
+ Load testing tool: Locust
+ API:
- Language: NodeJs (v12)
- Framework: NestJs (v7.6)
- ORM: Typeorm (v0.2.29)
- Server specs: 4CPU/4GB RAM (x2)
- Other: run as docker container
- OS: Ubuntu 20
- AVG CPU/RAM during testing: ~30%
+ Database:
- RDBMS: MySql (v8)
- Server specs: 8CPU/8GB RAM (x2)
- Other: run in replication mode (1 Master, 1 Slave)
- OS: Ubuntu 20
- AVG CPU/RAM during testing:
master: ~10%
slave: ~20%
+ Load balancer
- Load balancer: HAProxy (v2.2.9)
- Server specs: 4CPU/8CPU RAM (x2)
- OS: Ubuntu 20
- AVG CPU/RAM during testing: ~10%
Please advise me on any suspect issue that could happen to be a bottleneck.