I have an SQL table that is around 1TB of data, I want to ETL this table to GCS. What I don't understand is how Apache beam reads the table, is it in chunks, if so what is the size of the chunk and how can I configure it? Also, does the job starts from where it left in case of failer?
1 Answers
Apache Beam provides the JdbcIO transform factory class to read data from JDBC accessible databases. JdbcIO uses normal JDBC constructs to establish connectivity and read data.
JdbcIO provides no mechanism to chunk, shard, or split it's workload. The JdbcIO.Read transform is created with a SQL query and that query is executed in a single step.
If you need to partition the workload to obtain some level of parallelization, that will need to be added to the Apache Beam pipeline. This can be accomplished the having an initial step the computes the partition boundaries and then passes those boundaries as inputs to the JdbcIO.Read transform's SQL query.
https://nl.devoteam.com/en/blog-post/querying-jdbc-database-parallel-google-dataflow-apache-beam/ is an example of this partitioning approach.
Related