I'm seeing an intermittent connection timeout when trying to establish a connection from a Heroku one-off dyno to a MySQL RDS instance over Heroku's Private Space Peering Connection. The error is a ETIMEDOUT error using the node mysql2 library. It occurs at a frequency of about 1/100 attempts with no pattern as to time of day. It's also worth noting a few of the following:
- This is a staging environment and has minimal traffic so the RDS metrics are mostly flat. Resource contention is likely not the issue. CPU hovers at 1%, IOPS are negligible, connection count is <10, storage and memory are over-provisioned.
- This error only started happening when I switched to using the peering connection as opposed to connecting to the database over the public internet.
- I do not see this error when using connection pooling on longer running processes.
- MySQL error logs are clean
These are the potential issues I've investigated and ruled out:
- VPC routes exist for all RDS subnets using 10.0.0.0/16 (Heroku CIDR) via the correct peering connection
- Network ACLS applied at subnet level are ANY/ANY for all RDS subnets
- RDS allows ingress traffic via 10.0.0.0/16 (Heroku CIDR)
- DNS resolution works across the peering connection
The current MySQL connection configuration is using the default 10000 connection timeout. I'm hesitant to raise this without understanding the root cause because 10s should be plenty to establish a connection. Here is the code for connecting and tearing down the connection:
const mysql = require('mysql2/promise');
// Heroku dyno boots up via the scheduler add-on
const connection = await mysql.createConnection({
timezone: '+00:00',
dateStrings: ['DATE', 'DATETIME'],
decimalNumbers: true,
host: process.env.MYSQL_HOST,
database: process.env.MYSQL_DATABASE,
user: process.env.MYSQL_USER,
password: process.env.MYSQL_PASSWORD,
port: parseInt(process.env.MYSQL_PORT, 10),
ssl: fs.readFileSync('...', 'utf8')
});
// Do some work
await connection.end();
// Heroku dyno is torn down
It's difficult to get insight into the Heroku private space side of things because it's so abstracted but has anyone seen this kind of issue?
UPDATE 1
Was able to get a few tools installed on the one-off dynos and could successfully telnet and connect with the mysql client DESPITE getting a timeout from the node mysql lib. So it looks like the node mysql2 lib is timing out while running these commands do not. Any reason why that would be the case?
# prints Connected to [HOST]. Escape character is '^]'.
$ echo -e '\x1dclose\x0d' | telnet ${process.env.MYSQL_HOST} ${process.env.MYSQL_PORT}
# prints 8.0.17
$ echo "SELECT VERSION() AS version" | mysql -h ${process.env.MYSQL_HOST} -u ${process.env.MYSQL_USER} -p${process.env.MYSQL_PASSWORD}
UPDATE 2
I've combed through flow logs for the RDS ENI around when we see a timeout and I see traffic coming from the IP of the machine that experienced the timeout. That confirms the traffic from debug telnet <HOST> <PORT> and mysql -h <HOST> ... commands are going through correctly.