Add SDF to pipeline without having to pass dummy input

user_725937 0
Viewed 89

I have a SDF written in Python:

class Foo(DoFn):
    def process(element: Pipeline) -> Iterable[str]:
        ...

According to docs SDFs are for implementing IO. For my IO I do not need to have input elements passed to the DoFn. The only option to add the SDF to the pipeline seems to be a ParDo:

p | ParDo(Foo())

Doing it this way fails since it expects a windowing, which is not present. Even when manually adding windowing it still later on fails when making other assumptions.

So how do I write a SDF without any input element and add it directly to the Pipeline?

apache-beam python
1 Answers

You can feed in a single record just to trigger that gets ignored by the Splittable DoFn.

Alternatively, you can try to use the Impulse transform.

user_46960150
Related