How to diagnose extremely slow AWS RDS MySQL Performance?

Question

My DB has around 15 tables, each with 40 columns, with 10.000 rows each. Most of it with VARCHAR, some indexes and foreign keys.

Sometime I need to reconstruct my database (design flaw, working on it), which takes about 40 seconds locally. Now I'm trying to do the same to a AWS RDS MySQL 5.75 instance, but it takes forever, something like 40-50 minutes. The last time I had to do this same process it took no more than 5 minutes, still way more than the local 40 seconds, but I'm happy with it.

My internet speed is at about 35 Mbps Download / 5 Mbps Upload. I know it's not fast, but it's consistent, and it hasn't changed since my last rebuilt.

I enabled General Logs, but all I can see are the INSERT queries, occasionally some "SELECT 1". I do have same space for improvements on my code, but still, from 00:40:00 to 50:00:00, it seems that there's something else going on.

Any ideas on how to diagnose and find the bottleneck? Thanks

--

Additional relevant information: It is a Micro instance from AWS, all of the relevant monitoring indicators are basically flat: CPU at 4%, Free Storage Space at 20.000 MB, Freeable Memory at 200 MB, Write IOPS at around 2,5, the server runs a 5.7.25 MySQL, 1vCPU, 1Gb of RAM and 20GB of SSD. This is the same as 3 months ago when I last rebuilt the database.

SHOW GLOBAL STATUS: https://pastebin.com/jSrAzYZP SHOW GLOBAL VARIABLES: https://pastebin.com/YxD7dVhR SHOW ENGINE INNODB STATUS: https://pastebin.com/r5wffB5t SHOW PROCESS LIST: https://pastebin.com/kWwiyGwf SELECT * FROM information_schema...: https://pastebin.com/eXGBmetP

I haven't made any big changes to the server configuration, except enabling logs, e maxing out max_allowed_packets and saving logs to file.

In my backend I have a Flask app running, when it receives the API call, it takes a bunch of pickled objects and adds them all to the database (appending the Flask SQLAlchemy class to a list) and then running db.session.add_all(entries), trying to run a bulk operation. The code is the same, both for localhost and my remote server.

It does get slower in three specific tables, most of them with VARCHAR columns, but nothing different from my last inserts - it seems odd that the problem would be data, or the way the code is structured, or at least doesn't seem reasonable that this would result in a 20 second (localhost) to 40 minutes (hosted server) time, specially when the rest of the tables work mostly the same.

user_1842677 · Answer

Enable the slow log, set long_query_time=0, run your code, then put the resulting log through mysqldumpslow.

Establish which queries contribute most to slowness and take it from there.

Compare the config between your old server and your new one.

Also, are they the same version of MySQL? 5.6, 5.7 and 8.0 can produce very different execution plans (with 5.6 usually coming up with the sane one if they differ).

user_4420649 · Answer

Rate Per Second = RPS

Suggestions to consider for your AWS RDS Parameters group

thread_cache_size=24  # from 8 to reduce threads_created count
innodb_io_capacity=1900  # from 200 to enable more use of SSD IOPS capacity
read_rnd_buffer_size=128K  # from 512K to reduce handler_read_rnd_next RPS of 21
query_cache_size=0  # from 1M since you have QC turned off with query_cache_typ=OFF

Determine why com_flush is running 13 times per hour and get it stopped to avoid table open thrashing.

user_8323793 · Answer

I found that after migrating to RDS all my database Indexes are gone! They weren't migrated along with the schema and data. Make sure you're indexes are there. Also, MySQL query cache is OFF by default in RDS. This won't help the performance of your initial query, but it may speed things up in general. You can set query_cache_type to 1 and define a value for query_cache_size. I also changed the thread_cache_size from 8 to 24 and innodb_io_capacity from 200 to 1900 don't know if it helps you. Also creating AWS DB Parameter Groups helped me a lot with configuring and tuning DB variables. Here you can read more: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html

How to diagnose extremely slow AWS RDS MySQL Performance?

3 Answers