How to optimize a sequential insert on mysql

user_6950600 0
Viewed 625

I'm trying to implement an HTTP event streaming server using MySQL where Users are able to append an event to a stream (a MySQL table) and also define the expected sequence number of the event.

The logic is somewhat simple:

  1. Open transaction
  2. get the next sequence number in the table to insert
  3. verify if the next sequence number matches the expected(if supplied)
  4. insert in database

Here's my code: 

    public async append(
        data: any = {},
        expectedSeq?: number
    ): Promise<void> {
        let published_at = $date.create();

        try {
            await $mysql.transaction(async trx => {
                let max = await trx(this.table)
                    .max({
                        seq: "seq",
                    })
                    .first();

                if (!max) {
                    throw $error.InternalError(`unexpected mysql response`);
                }

                let next = (max.seq || 0) + 1;

                if (expectedSeq && expectedSeq !== next) {
                    throw $error.ExpectationFailed(
                        `expected seq does not match current seq`
                    );
                }

                await trx(this.table).insert({
                    published_at,
                    seq: next,
                    data: $json.stringify(data),
                });
            });
        } catch (err) {
            if (err.code === "ER_DUP_ENTRY") {
                return this.append(data, expectedSeq);
            }

            throw err;
        }
    }

My problem is this is extremely slow since there are race conditions between parallel requests to append to the same stream.. my laptop inserts/second on one stream went from ~1k to ~75.

Any pointers/suggestions on how to optimize this logic?

CONCLUSION

After consideration from comments, I decided to go with auto increment and reset the auto_increment only if there's an error. It yields around the same writes/sec with expectedSeq but much higher rate if ordering is not required.

Here's the solution:

    public async append(data: any = {}, expectedSeq?: number): Promise<Event> {
        if (!$validator.validate(data, this.schema)) {
            throw $error.ValidationFailed("validation failed for event data");
        }

        let published_at = $date.create();

        try {
            let seq = await $mysql.transaction(async _trx => {
                let result = (await _trx(this.table).insert({
                    published_at,
                    data: $json.stringify(data),
                })).shift();

                if (!result) {
                    throw $error.InternalError(`unexpected mysql response`);
                }

                if (expectedSeq && expectedSeq !== result) {
                    throw $error.ExpectationFailed(
                        `expected seq ${expectedSeq} but got ${result}`
                    );
                }

                return result;
            });

            return eventFactory(this.topic, seq, published_at, data);
        } catch (err) {
            await $mysql.raw(`ALTER TABLE ${this.table} auto_increment = ${this.seqStart}`);

            throw err;
        }
    }
knex.js typescript mysql sql
4 Answers

Why does the web page need to provide the sequence number? That is just a recipe for messy code, perhaps even messier than what you sketched out. Simply let the auto_increment value be returned to the User.

  1. INSERT ...;
  2. SELECT LAST_INSERT_ID(); -- session-specific, so no need for transaction.
  3. Return that value to user.
user_17668310

Why not use Apache Kafka, It does all of this natively. With the easy answer out of the way, optimization is always tricky with partial information, however I think you've given us one hint that might enable a suggestion. You said without the order clause it performs much faster, which means that getting the max value is what is taking so long. That tells me a few things, first this value is not the clustered index (which is good news), second that you probably do not have sufficient index support (also good news since it's fixable by creating an index on this column, and sorting the index desc). This sounds like a table with millions or billions of rows in it, and this particular column has no guaranteed order, without the right indexing you could be doing a table scan between inserts to get the max value.

user_55277520

Why not use a GUID for your primary key instead of an auto-incremented integer? Then your client could generate the key and would also be able to insert it every time for sure.

user_2181770

Batch inserts versus singleton inserts

Your latency/performance problem is due to a batch size of 1 - as each send to the the database requires multiple round trips to the rdbms. Rather than inserting one row at a time, with a commit and verification after each row, you should rewrite your code to issue batch sizes of 100 or 1000 at a time, inserting n rows and verifying per batch rather than per row. If the batch insert fails, you can retry one row at a time.

user_42982080
Related