Is there a way to update side inputs in Apache Beam?

user_1255620 0
Viewed 822

I am developing a data transformation pipeline in Apache Beam, where I need some look up table to help with transforming each incoming record.

I can pass in the look up table as a side input, but the caveat is an incoming record could update the lookup table that I am using.

Is there a way to update the lookup table and then broadcast the update to every other worker?

Update: One possibility is to use a data driven trigger to signal an end to the current window when an incoming record updates the side input. Does side input gets automatically refreshed at the start of the next window? In my case I am retrieving the side input from an external source.

apache-beam
1 Answers

If the lookup table is related to the key and window you can try using state https://beam.apache.org/blog/2017/02/13/stateful-processing.html

Otherwise you may need an external data storage. Database or in memory cache. Just be aware of the fact that DoFns are serializable and opening/closing connections need to be done carefully.

user_2219510
Related