Confluent:Topic was not created after loading a Connector

user_7987546 0
Viewed 1225

I tried connecting to several databases including MySQL and MSSQL and i had no problem.

But when i tried to connect to a certain remote MySQL database (That can be accesses only from my company's network) which contains a view that has almost 3 million records, the Connector was loaded and the Status said it's running but there wasn't any topic created to consume the data from it.

What could be the reason? and where can i find the correct log file to know what happened?

Connector Status

Available Topics

Here is an example of what the connector looks like:

{
  "name": "mysql-source",
   "config": {
   "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
   "key.converter": "io.confluent.connect.avro.AvroConverter",
   "key.converter.schema.registry.url": "http://localhost:8081",
   "value.converter": "io.confluent.connect.avro.AvroConverter",
   "value.converter.schema.registry.url": "http://localhost:8081",
   "incrementing.column.name": "Id",
   "tasks.max": "1",
   "table.types": "VIEW",
   "table.whitelist": "ticket_rep",
   "mode": "incrementing",
   "topic.prefix": "mysql-",
   "name": "mysql-source",
   "validate.non.null": "false",
   "connection.url": "jdbc:mysql://XX.XXX.XX.XX:3306/database? 
    user=user&password=password"
 }

}

These are the log results when i run confluent log connect:

> [2018-09-11 16:37:57,382] ERROR Failed to run query for table TimestampIncrementingTableQuerier{name='foo', query='null', topicPrefix='mysql-', timestampColumn='null', incrementingColumn='id'}: {} (io.confluent.connect.jdbc.source.JdbcSourceTask:247)
java.sql.SQLException: Java heap space
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:975)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1025)
    at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.executeQuery(TimestampIncrementingTableQuerier.java:201)
    at io.confluent.connect.jdbc.source.TableQuerier.maybeStartQuery(TableQuerier.java:84)
    at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.maybeStartQuery(TimestampIncrementingTableQuerier.java:55)
    at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:225)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:179)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2018-09-11 16:38:02,523] ERROR Failed to run query for table TimestampIncrementingTableQuerier{name='foo', query='null', topicPrefix='mysql-', timestampColumn='null', incrementingColumn='id'}: {} (io.confluent.connect.jdbc.source.JdbcSourceTask:247)
com.mysql.cj.jdbc.exceptions.PacketTooBigException: Packet for query is too large (7,562,612 > 4,194,304). You can change this value on the server by setting the 'max_allowed_packet' variable.
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:107)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:975)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1025)
    at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.executeQuery(TimestampIncrementingTableQuerier.java:201)
    at io.confluent.connect.jdbc.source.TableQuerier.maybeStartQuery(TableQuerier.java:84)
    at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.maybeStartQuery(TimestampIncrementingTableQuerier.java:55)
    at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:225)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:179)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
confluent-platform apache-kafka-connect apache-kafka jdbc
1 Answers

com.mysql.cj.jdbc.exceptions.PacketTooBigException: Packet for query is too large is a problem on MySQL side which can be fixed by increasing the value of max_allowed_packet variable. To do so, you need to include

max_allowed_packet=512M

under my.cnf (or my.ini, depending on the OS you are running) file(within [mysqld] section) and then restart MySQL. After restarting MySQL, 

SHOW VARIABLES LIKE 'max_allowed_packet';

should return the value you've set in your MySQL config file.For more details regarding this error you can refer to the MySQL documentation.


java.sql.SQLException: Java heap space, indicates that Kafka connect is running out of heap space. It is possible to control starting and max heap size by running 

KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties mysql-source-connector.properties

that sets a starting heap size of 512 MB and a maximum size of 1 GB. You might need to change the sizes, according to your needs.

user_71317570
Related