Error: Handshake inactivity timeout (MySQL + Node.js)

Viewed 2811

Using docker-compose with ngnix, node backend and mysql.js. This error has occurred:

api      | there was an error connecting to pool:  Error: Handshake inactivity timeout
api      |     at Handshake.<anonymous> (/app/node_modules/mysql/lib/protocol/Protocol.js:160:17)
api      |     at Handshake.emit (node:events:376:20)
api      |     at Handshake._onTimeout (/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:124:8)
api      |     at Timer._onTimeout (/app/node_modules/mysql/lib/protocol/Timer.js:32:23)
api      |     at listOnTimeout (node:internal/timers:556:17)
api      |     at processTimers (node:internal/timers:499:7)
api      |     --------------------
api      |     at Protocol._enqueue (/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
api      |     at Protocol.handshake (/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
api      |     at PoolConnection.connect (/app/node_modules/mysql/lib/Connection.js:116:18)
api      |     at Pool.getConnection (/app/node_modules/mysql/lib/Pool.js:48:16)
api      |     at Pool.query (/app/node_modules/mysql/lib/Pool.js:202:8)
api      |     at /app/db/connect.js:18:10
api      |     at new Promise (<anonymous>)
api      |     at Object.connectionFunc (/app/db/connect.js:17:21)
api      |     at Object.<anonymous> (/app/routes/index.js:17:12)
api      |     at Module._compile (node:internal/modules/cjs/loader:1108:14) {
api      |   code: 'PROTOCOL_SEQUENCE_TIMEOUT',
api      |   fatal: true,
api      |   timeout: 10000
api      | }

As another post mentioned, I tried changing the timeout and that didn't work.

I have to use a network to connect my docker-compose files. I've intermittently had it working without using an internal network, but other times I get the following message:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

This itself is strange as there are only the 3 default networks (and one I have created). Other posts have said that if you go over 30 or more networks this should become an issue -

root@ubuntu-s-1vcpu-1gb-nyc1-01:~/blog/blog# docker network ls
NETWORK ID     NAME       DRIVER    SCOPE
430e230e6641   bridge     bridge    local
915d73adee82   host       host      local
8cd9e79fd644   none       null      local
9f936dc688b7   underdev   bridge    local

So i.e. I've tried this in my docker-compose.yml

  db:
    container_name: db
    build: ./db
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=True
      - MYSQL_DATABASE=secret
      - MYSQL_USER=secret
      - MYSQL_ROOT_PASSWORD=secret
      - MYSQL_PASSWORD=secret
    volumes:
      - db_data:/var/lib/mysql
    networks:
      - underdev
volumes:
  db_data: {}
networks:
 underdev:
  external: true

I've also tried this:

networks:
  default:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.57.0/24

Both fix the ipv4 error, but then I still get the handshake inactivity timeout error. Someone suggested using network_mode: bridge, but shouldn't the network use the default bridge automatically? If anyone has any ideas please let me know.

PS:

Here is the connection to the db - I tried adding a timeout to the connection handler - the timeout worked, but the connection still didn't have a handshake.

var mysql = require('mysql');
  var testconnection = mysql.createConnection({
    host: 'db',
    port: '3306',
    user: 'root',
    password: process.env.MYSQL_ROOT_PASSWORD,
    database: 'blog',
    debug: true
  })
  var pool  = mysql.createPool({
    connectionLimit : 100,
    host            : 'db',
    port            : '3306',
    //connectTimeout  : 20000,
    //acquireTimeout  : 20000,
    //timeout         : 20000,
    user            : 'root',
    password        : process.env.MYSQL_ROOT_PASSWORD,
    database        : 'blog'
  });
  console.log("-----------------------------------------------------");
  console.log("value of testconnection: ", testconnection);
  console.log("-----------------------------------------------------");
  testconnection.query("SELECT * from blogs", function(err, rows, fields){
    console.log("value of err: ", err);
    console.log("value of rows: ", rows);
    console.log("value of fields: ", fields);
  })
  console.log("-----------------------------------------------------");
  console.log("value of pool: ", pool);
  console.log("-----------------------------------------------------");
  let poolPromise = new Promise(resolve=>{
    pool.query("SELECT * FROM blogs", function(err,rows,fields){
      /*
      console.log("value or err: ", err);
      console.log('value of rows: ', rows);
      console.log('value of fields: ', fields);
      */
      if (err){
        console.log('there was an error connecting to pool:', err);
        process.exit();
        resolve(false);
      }else{
        resolve(pool);
      }
    })
  })

  return poolPromise;
}

module.exports = {
  connectionFunc
}
         

EDIT:

I've written scripts to completely uninstall and completely reinstall docker (including all folders and subdirectories) and I still have the following error message:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Here is script to remove all of docker:

#!/bin/bash
  
sudo apt-get purge -y docker-engine docker docker.io docker-ce docker-ce-cli
sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce

sudo apt-get remove docker docker-engine docker.io containerd runc

sudo rm -rf /var/lib/docker /etc/docker
sudo rm /etc/apparmor.d/docker
sudo groupdel docker
sudo rm -rf /var/run/docker.sock

And here is the script I use to install docker (which I confirm works with the hello world test) and comes from commands in this page of the documentation (https://docs.docker.com/engine/install/ubuntu/):

#!/bin/bash
  

echo 'apt-get update'
echo '-------------------------------------------------------------------------------'
sudo apt-get update
echo '-------------------------------------------------------------------------------'
echo 'sudo apt-get install'
echo '-------------------------------------------------------------------------------'
yes | sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
echo 'curl software'
echo '-------------------------------------------------------------------------------'
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
echo '-------------------------------------------------------------------------------'
echo 'add-apt-repository'
echo '-------------------------------------------------------------------------------'
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
echo 'sudo apt-get update'
echo '-------------------------------------------------------------------------------'
sudo apt-get update
echo '-------------------------------------------------------------------------------'
echo 'install docker-ce-cli'
echo '-------------------------------------------------------------------------------'
yes | sudo apt-get install docker-ce docker-ce-cli containerd.io
echo '-------------------------------------------------------------------------------'

EDIT:

Adding a links variable to tha api container in the docker-compose file didn't fix the issue:

    networks:
      - underdev
    links:
      - db:db

EDIT:

Tried rebooting the server. That didn't work, same error. I've also tried upgrading the server via

sudo apt-get upgrade && sudo apt-get dist-upgrade

and running the application again - same error.

I'm currently running 18.04.5 ubuntu bionic:

root@ubuntu-s-1vcpu-1gb-nyc1-01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

Tried upgrading to 20.04.1 and that also doesn't work.


1 Answers

I solved the problem in another way. It wasn't an inbound issue. I'm using 'Standard TCP/IP over SSH' in order to connect Mysql-server in AWS. There is no problem normally. But suddenly it happended.

"Handshake inactivity timeout"!!

I found out that another application is binding port 3306 of the development environment. You can use command 'netstat -anb'. After deleting it and reconnecting 'Standard TCP/IP over SSH' again, the problem went away.

Related