Pentaho Table input to table output in batches

user_2772061 0
Viewed 5534

My input table in MySQL has 20 Million records and the target table in Oracle is empty. I need to load the entire table from MySQL into Oracle. I am simply using a Table Input and Table Output step.

PentahoProcessing

My intention is not to lock the source table for a long time whilst reading. Is there a problem with the load (Number of records) I am trying to achieve? I could see Use batch update for inserts option in the Table Output. I could not see something similar in the Table Input. Is there a way to perform batch processing in Pentaho?

kettle pentaho transformation mysql
2 Answers

In additinal of @AlainD solution There are a couple of options: - Tune MySQL for better performance on Inserts - Use the MySQL Bulk loader step in PDI - Write SQL statements to file with PDI and read them with mysql-binary

Speed can be boosted by using some simple JDBC-connection setting.

  • useServerPrepStmts= false
  • rewriteBatchedStatements= true
  • useCompression= true
user_13817370
Related