Strange intermittent issue with MariaDB Connector/J and AWS RDS Aurora

Viewed 446

Config goes as follows:

  • AWS RDS Aurora cluster, single region, single master, two read replicas
  • Java nodes running within the same VPC
  • MariaDB Connector/J with Aurora failover mode
  • HikariCP as the connection pool
  • HikariCP has the default maxLifetime of 30 minutes
  • Aurora has the default wait_timeout of 8 hours

In order to instruct MariaDB Connector/J to execute a given query against an Aurora read replica vs master, I'm calling Connection.setReadOnly(true) after obtaining it from HikariDataSource.

Everything works well except... about twice a week on average, 2% of the Java node fleet will hit an exception while trying to query a read replica. The exceptions occur on the different nodes within a second or two from one another, against the same replica.

The underlying exception stack looks as follows:

Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=2445137) Communications link failure with secondary host xxx.xxx.us-east-1.rds.amazonaws.com:3306. (conn=1644010) Connection is closed
      at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:66)
      at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:153)
      at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:273)
      at org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:229)
      at org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:149)
      at org.mariadb.jdbc.ClientSidePreparedStatement.executeQuery(ClientSidePreparedStatement.java:163)
      at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
      at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
      at com.querydsl.sql.AbstractSQLQuery.iterateSingle(AbstractSQLQuery.java:370)
      ... 21 more
Caused by: java.sql.SQLException: Communications link failure with secondary host xxx.xxx.us-east-1.rds.amazonaws.com:3306. (conn=1644010) Connection is closed
on HostAddress{host='xxx.xxx.us-east-1.rds.amazonaws.com', port=3306},master=false.  Driver has reconnect connection
      at org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:563)
      at org.mariadb.jdbc.internal.failover.FailoverProxy.handleFailOver(FailoverProxy.java:391)
      at org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(FailoverProxy.java:324)
      at org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(FailoverProxy.java:294)
      at com.sun.proxy.$Proxy27.executeQuery(Unknown Source)
      at org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:220)
      ... 26 more

Next, if I review the MySQL error log for that Aurora instance, it will report aborting the connections seconds or sometimes a minute after the exceptions occur on the Java nodes:

1644010 [Note] Aborted connection 1644010 to db: 'xxx' user: 'xxx' host: 'xxx' (Got an error reading communication packets)

Has anyone run into anything similar? Any ideas as to what could be causing this?

0 Answers
Related