Use debezium link postgresql 11 Couldn't obtain encoding for database test

user_11105152 0
Viewed 1952

I use debezium cdc connect pg, and i build the pg 11 use by docker，the pg is run well. when i use debezium in kafka connector， it report：

Couldn't obtain encoding for database test

the curl is:

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST http://localhost:8083/connectors/ -d '{
    "name": "debezium",
    "config": {
        "name": "debezium",
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "localhost",
        "database.port": "5432",
        "database.dbname": "test",
        "database.user": "pg",
        "database.password": "135790",
        "database.server.name": "ls",
        "table.whitelist": "public.test",
        "plugin.name": "pgoutput"
    }
}'

the kafka exception is:

[2020-07-08 09:24:35,076] ERROR Uncaught exception in REST call to /connectors/ (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
java.lang.RuntimeException: Couldn't obtain encoding for database test
    at io.debezium.connector.postgresql.connection.PostgresConnection.determineDatabaseCharset(PostgresConnection.java:434)
    at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:77)
    at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:87)
    at io.debezium.connector.postgresql.PostgresConnector.validate(PostgresConnector.java:102)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:277)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:534)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:531)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:267)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:216)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: FATAL: database "test" does not exist
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
    at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2644)
    at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:137)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:255)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
    at org.postgresql.Driver.makeConnection(Driver.java:458)
    at org.postgresql.Driver.connect(Driver.java:260)
    at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:190)
    at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:788)
    at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:783)
    at io.debezium.connector.postgresql.connection.PostgresConnection.determineDatabaseCharset(PostgresConnection.java:431)
    ... 13 more
[2020-07-08 09:24:35,128] INFO 127.0.0.1 - - [08/Jul/2020:01:24:34 +0000] "POST /connectors/ HTTP/1.1" 500 73  330 (org.apache.kafka.connect.runtime.rest.RestServer:60)
debezium apache-kafka-connect apache-kafka postgresql
3 Answers

it seems to me that the database named test either does not exist or is not visible by the user pg.

user_85725860

Couple of things that are inaccurate in your payload. Keep resolvable name instead of localhost. Secondly, place correct db namespace.

        "database.hostname": "FQDN",
        "database.server.name": "test_table_name",

It may be the case where, PostgreSQL host has not validated the authentication with pg user. It needs to have an entry into pg_hba.conf (on PostgreSQL server) and establish trust/auth from the client machine, i.e. Kafka-connector.

# host       DATABASE  USER  ADDRESS                       METHOD  [OPTIONS]
# hostssl    DATABASE  USER  ADDRESS                       METHOD  [OPTIONS]
host         test      pg    Kafka.connector.server.ip/32  md5
hostssl      test      pg    Kafka.connector.server.ip/32  md5

Then, restart the PostgreSQL server to get auth for pg user in effect, in my case pg_ctl reload.

As, the curl happens as a REST API calls, get Kafka-rest(8082) and Kafka-connect-rest(8083) ports added in the firewall settings of PostgreSQL server.

user_146297370

Yea, the message is differently misleading. In my case the problem was in closed ports between kafka-connect and database server.

user_12182480
Related