How can I modify the apache beam DirectRunner to make it faster?

Viewed 399

We are building a framework using apache beam, and there are some use cases (like late data) that we simply don't need. What we do need however, is blazing-fast performance.

I'm trying to modify the DirectRunner to be faster. So far I've commented out Metrics and Enforcements, which yielded a small performance increase. We've also removed SynchronizedProcessingTimeOutputWatermark and SynchronizedProcessingTimeInputWatermark without affecting our use case, which suggests that there may be more (Watermarks? Holds?) which could be removed.

I realise this is a rather vague question, but what else can I remove to make it faster?

2 Answers

You could also consider using a "production" runner such as Flink in local mode.

Related