Bulk ops in Apache beam Java with native throttling

user_1643558 0
Viewed 80

We have multiple spreadsheets. each has 2 to 4 sheets that describe entities. rows in sheet 1 map to rows in sheet 2 and so on.

Can use APACHE Poi to read and map these to my POPJOs. I tried with Open CSV but that has CSV support and not POI as a reader?

Also when processing this data, need to call an API with attachments. We were planning to use Apache beam direct runner, and able to do a POC. only thing is that not sure how to throttle the processes. so only 1 record is processed every minute?

We have 4 attributes to look at : job product, tenant, sub type and priority

Would like limits at product, tenant and sub type level. Meaning even if a tenant can have 10 processes, if the product has a limit of 15 and there are already 15 of other tenants running, then make this wait. We can use a SINGLE_BEAM for 1 upload but how to limit across all jobs?

We have a classes to represent jobs, tasks (rows inside a file or some other sub unit of work), tenants, to track status and counts

We can write our own Java code to enforce this, but not sure where to plug this in to Apache Beam ?

apache-beam pipeline java
1 Answers

There isn't a built-in I/O for Apache POI: https://beam.apache.org/documentation/io/built-in/.

You can follow this instruction to create a customize source https://beam.apache.org/documentation/io/developing-io-java/ if needed.

You can also write some ParDo with stateful processing to control your throughput.

A state is per-key-per-window, so as long as you limit your pipeline to a single key (or desired number of "process"es as you mentioned) and using the default global window for batch, you can limit the parallelism.

user_31758890
Related