Apache Bean Spark Runner does not work on streaming mode - java.lang.IllegalAccessException

user_14451861 0
Viewed 164

I have a streaming beam application that runs on Flink. When I tried to switch it to spark runner with EMR (5.30.1), with both apache bean (2.23.0 and 2.24.0), I am getting following error:

Exception in thread "main" java.lang.IllegalAccessException: Class org.apache.spark.sql.streaming.DataStreamReader can not access a member of class org.apache.beam.runners.spark.structuredstreaming.translation.streaming.DatasetSourceStreaming with modifiers ""
    at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
    at java.lang.Class.newInstance(Class.java:436)
    at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161)
    at org.apache.beam.runners.spark.structuredstreaming.translation.streaming.ReadSourceTranslatorStreaming.translateTransform(ReadSourceTranslatorStreaming.java:71)
    at org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.applyTransformTranslator(PipelineTranslator.java:159)
    at org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.visitPrimitiveTransform(PipelineTranslator.java:214)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:664)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:656)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:656)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:656)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
    at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
    at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:463)
    at org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.translate(PipelineTranslator.java:173)
    at org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner.translatePipeline(SparkStructuredStreamingRunner.java:195)
    at org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner.run(SparkStructuredStreamingRunner.java:148)
    at org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner.run(SparkStructuredStreamingRunner.java:68)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
    at com.adp.vantage.sonic.pipelines.KafkaPipelineConsumer.main(KafkaPipelineConsumer.java:136)

The command line is:

spark-submit --class com.adp.vantage.sonic.pipelines.KafkaPipelineConsumer deploy/sonic-payroll-event-tracker-spark-2.10.0.jar --runner=SparkStructuredStreamingRunner --streaming=true --env=dit

The DatasetSourceStreaming is not a public class, so looks to me this will never work, does anybody manage to make it work at all?

apache-beam
1 Answers

Unfortunately, SparkStructuredStreamingRunner still doesn’t support streaming job (please, take a look on “Note" section here)

There is a Jira issue for tracking this - in two words, it’s blocked because of issue on Spark with multiple aggregations for Structured Streaming component.

So far, it’s recommended to use a “Classical” RDD or Portable Spark runners to run streaming Beam pipelines on Spark.

user_106873250
Related