Kotlin + JUnit + postgres for integration testing: leaked connections

Question

I set up a test class that runs 13 integration tests. All of the tests involve 1 database, 1 schema and 3 tables in the schema. The test class is set up so that I create the test database with @BeforeClass and delete the test database with @AfterClass. In the setUp phase for each test, I migrate the schema and populate the tables with artificial data. Then in the tearDown phase I remove the schema. In each test, I run my test logic against the test database:

class MyTest {
  companion object {
    private val helper = MyHelper()
    @BeforeClass @JvmStatic
    fun createTestDatabase() {
        helper.createTestDatabase()
    }
    @AfterClass @JvmStatic
    fun removeTestDatabase() {
        helper.deleteTestDatabase()
    }
  }
  @Before
  fun setUp(){
    // migrate the schema and populate the tables with artificial data
    helper.migrate()
    helper.populate()
  }

  @After
  fun tearDown() {
    // remove the schema
    helper.removeSchema()
  }
  @Test MyFirstTest() {
    // populate the tables with artificial data
    ... // test logic
  }
  ...
}

The database manipulations are all implemented in MyHelper class. In particular, I have one function in MyHelper that executes a query string:

class MyHelper {
  protected val POSTGRES_USER = "postgres"
  protected val POSTGRES_HOST = "localhost"
  protected val POSTGRES_PORT = 5432
  protected val POSTGRES_TEST_DATA_DATABASE = "my_database_test"
  protected val POSTGRES_TEST_DATA_SCHEMA = "my_schema"
  protected val POSTGRES_DEFAULT_DATABASE = "postgres"
  protected val POSTGRES_DEFAULT_SCHEMA = "postgres"

  protected val graphDataTables = listOf("my_table_1", "my_table_2", "my_table_3")

  val databases = listOf(POSTGRES_DEFAULT_DATABASE, POSTGRES_TEST_DATA_DATABASE)
  val dataSources = databases.associateBy({it}, { getDataSource(it)})

  fun getDataSource(database: String): PGSimpleDataSource {
      val ds = PGSimpleDataSource()
      val url = "jdbc:postgresql://$POSTGRES_HOST:$POSTGRES_PORT/$database"
      ds.setUrl(url)
      ds.setUser(POSTGRES_USER)
      return ds
  }

  fun executeQuery(sqlQuery: String,
                   database: String = this.POSTGRES_DEFAULT_DATABASE,
                   schema: String = this.POSTGRES_DEFAULT_SCHEMA
  ): Unit {
      val ds = dataSources[database]!!
      ds.currentSchema = schema

      ds.connection.use {
          it.createStatement().use {
              it.execute(sqlQuery)
          }
      }
  }
  ...
}

The function MyHelper.populate() uses the executeQuery function to insert data into the tables.

My problem is that it seems that I have many unclosed connections (leaked connections). If I keep the # of tests under 10, then all tests pass without a problem. As soon as I pass 10, I get errors like the following for tests 11 and up:

Nov 09, 2017 11:31:55 AM org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
    at org.postgresql.Driver$ConnectThread.getResult(Driver.java:401)
    at org.postgresql.Driver.connect(Driver.java:259)
    ...

for the tests that got run after the 10th. And the last stack in the log points to the ds.connection.use { line in the function executeQuery.

However, if I move the helper.createTestDatabase() out of the companion object, and put it in the init block for MyTest, like so:

class MyTest {
  companion object {
    private val helper = MyHelper()
  }
  init {
    helper.createTestDatabase()
  }
  ...
}

then I don't get any error even if the # of my tests are over 10. The problem I see with this is that now I have a test database created in my postgres docker container and it still exists after I'm done with the integration tests. I have no idea how to call removeTestDatabase now.

I run my tests in debug mode and stopped in the middle of the 10th test, then with

select * from pg_stat_activity;

I can see that there are many idle connections (almost 100) in the first scenario. In the 2nd scenario, I have only 10 idle connections in the middle of the 10th test (I'd like the number of idle connections to be 0, of course). My understanding is that my default postgres setting is that max_connections is set to 100. That's why I get the error when I have more than 10 tests in the first scenario. But I have no idea how I am creating idle connections.

Kotlin + JUnit + postgres for integration testing: leaked connections

0 Answers