Is it possible to collect mysql queries from independent rest api requests (to a php server) and run a combined insert or update?

Viewed 41

I have a use case where a large number of independent nodes are sending http requests with data packets every minute. I have to update a mysql table with data and time stamp from each packet.

Currently, I handle each api request independently, do a IODKU statement. The number of requests are increasing (as in ~100k requests per minute, and counting) and I am looking into alternate methods.

Is it even possible to save all these queries from a minute to be collected and then run as a single query?

EDIT: Using up RDS IOPS is the face of the issue at present.

1 Answers

It depends how strict your requirement is for durability. It also depends on if your traffic has bursts of high rates of traffic, or if it's fairly continual in having a high rate of traffic.

If you need for every request to result in a durable change, then you must handle the database write (IODKU) during the request, and it must be committed by the time you send a response for that request.

Writes to a durable database are pretty costly, and there's an upper limit to the amount of throughput a single server can sustain.

One strategy is to reduce the number of durable writes.

If you can tolerate some limited amount of non-durability (i.e. there's a small risk of data loss), then you could post those updates to a high-speed, in-memory key/value store (for example, Memcached or Redis), and send a success response to the client who sent the request.

Then once every interval of time, say every two seconds, copy the current value of each key in the key/value store to the durable database. This could help reduce the database writes by a few orders of magnitude, reducing it to a level that the database server type can sustain. I suppose it depends on the ratio of inserts of new keys versus updates of existing keys.

A second strategy is to scale out.

A single server has a finite capacity for sustained throughput of durable writes, it might be for example 5000 writes per second. So you could get two servers for 10,000 writes per second, or 20 servers for 100,000 writes per second. However many you need to handle the expected highest level of traffic.

You'd want to have your application code have some smarts to split up the writes approximately fairly over these servers, so none of them is overloaded while others are underutilized.

Both of these strategies are more complex to code than a beginner project that has one PHP app and one database server. Welcome to the big leagues! If you have such high volume of traffic, you must build an architecture that can handle it.

Related