flink impala jdbc insert is slow

user_16249609 0
Viewed 380

currently we're facing some performance issue for flink job using jdbc to insert around 1 millions data per hour to Kudu table using impala jdbc.

we've tried to increase the parameters

JdbcExecutionOptions.builder()
    .withBatchSize(1000)
    .withBatchIntervalMs(200)
    .withMaxRetries(3)
    .build()

with checkpoint interval 10 second and checkpoint timeout 10 minute with no result. we also try to increase the parallel from 3 to 6, but also not working.

we also try to increase the BatchSize to 10000 but we are facing error

org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat [] - JDBC executeBatch error, retry times = 0
java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Exceeded the statement exp
ression limit (250000)
Statement has 720000 expressions.

the best we could get is 1500 records / minute, with parameters of

JdbcExecutionOptions.builder()
    .withBatchSize(200)
    .withBatchIntervalMs(50)
    .withMaxRetries(3)
    .build()

and checkpoint interval 3 second and checkpoint timeout 10 minute.

Is there any others parameters that we could use to get better performance of this case ?

Thanks,

impala apache-flink jdbc
1 Answers

I don't know the exact config flink, so please provide full details. The most common error is due to lack of rewriteBatchedStatements=true in URL. That's because JDBC will send 1000 single insert statements instead of 1 consisting of 1000 insert statements.

user_96444860
Related