AWS RDS MySQL innodb/btr_search_latch

You're trying to send 50,000 push notifications in five minutes?

50,000 / 300 seconds means you're pushing 167 notifications per second and I assume then updating the database to record the result of the push. Probably you are doing this in many concurrent threads so you can do the pushes in parallel.

Have you considered doing these push notifications more gradually, like over 10 or 15 minutes?

Or updating the database in batches?

Or using fewer threads to avoid the high contention on the database?

I used to work for SchoolMessenger, a company that provide notification services for the majority of public schools in the USA. We sent millions of notifications, SMS messages, and phone calls every day. The way we did it was to have a very complex Java application queue up the notifications, and then post them gradually. Then as the results of the pushes came in, these also queued up, and updated the database gradually.

We used MySQL, but we also used it together with ActiveMQ as a persistent queue. Push all the tasks to be done into the queue, then a pool of worker threads would act on the tasks, and push the results back into another queue. Then a result-reading thread would read batches of results from the queue and update the database in bulk updates.

When you are designing a back-end system to do large-scale work to do, you have to think of new ways to architect your application to avoid choke-points.

As a database performance and scaling consultant, I have observed this rule many times:

For every 10x growth in data or traffic, you should reevaluate your software architecture. You may have to redesign some parts of it to work at the larger scale.