I have following Apache Beam pipeline
pipeline
.apply("Read lines", TextIO.read().from(options.getFileInput()))
.apply("Split lines", ParDo.of(new LineSplittableDoFn()))
.apply("Log bundles", ParDo.of(new ResultLogDoFn()));
- LineSplittableDoFn - Splittable DoFn implementation with @GetInitialRestriction, @ProcessElement with RestrictionTracker usage and @SplitRestriction.The DoFn splits the text line to 4 chunks.
- ResultLogDoFn - log unique bundles. Similar to code from https://www.waitingforcode.com/apache-beam/data-partitioning-apache-beam/read
Based on my observations:
- Only TextIO.read() controls amount of created bundles.
- LineSplittableDoFn - creates and splits initial restrictions based on implemented logic, I can see it from my logs.
- New elements are created by LineSplittableDoFn and runner has ability to do work stealing based on restrictions.
- Number of parallel executions stay same. I can't add more workers or worker threads because there are no new bundles created during restrictions splitting. Based on my understanding of documentation bundle is a unit of parallelization in Beam world so parallelism depends on it: https://beam.apache.org/documentation/runtime/model/#dependent-parallellism
The solution with Reshuffle works fine. I can see increase in number of bundles and parallelism.
Reshuffle.ViaRandomKey<ShardedRequest> shuffle = Reshuffle.viaRandomKey();
Reshuffle.ViaRandomKey<ShardedRequest> partitionedShuffle = shuffle.withNumBuckets(100);
pipeline
.apply("Read lines", TextIO.read().from(options.getFileInput()))
.apply("Split lines", ParDo.of(new LineNonSplittableDoFn()))
.apply(partitionedShuffle)
.apply("Log bundles", ParDo.of(new ResultLogDoFn()))
I'm using Datafow Runner (via flex templates), Apache Beam version is 2.24.0
Is it possible to increase number of bundles(and pipeline parallelism) using SplittableDoFn?