If I use Dataproc, how does it process real-time streaming data from Apache Hadoop and Spark to Dataproc?

Solution 1 :

I think you can use Spark Structured streaming on dataproc cluster with Cloud Pub Sub :

https://cloud.google.com/pubsub/lite/docs/write-messages-apache-spark

Previously Pub Sub was only compatible with Spark DStream but now you can use it with Spark structured streaming.

Solution 2 :

You can think about using Databricks on GCP, the product is young on GCP but offers many features on the top of Spark (for batch and streaming).

Solution 3 :

If it's not too hard to rewrite your jobs, you can also think about a migration from Spark to Beam with Dataflow runner.

Beam with Dataflow offers more features regarding Streaming and is serverless and fully managed on GCP.