Spark: sparkSession read from the result of an http response

user_10461625 0
Viewed 71

Small question regarding Spark and how to read from the result of a http response please.

It is well known Spark can take as datasource some database, or CSV, etc...

sparkSession.read().format("csv").load("path/to/people.csv");

sparkSession.read().format("org.apache.spark.sql.cassandra").options(properties).load()

May I ask how to read from the result of an http call directly please?

Without having to dump the data back inside another intermediate csv / intermediate database table.

For instance the csv and database would contains millions of rows, and once read, the job needs to perform some kind of map reduce operation.

Now, the exact same data comes from the result of an http call. It is small enough for the network layer, but the information contained inside the payload is big, so I would like to apply the same map reduce.

How to read from the response of an http call please?

Thank you

apache-spark java
2 Answers

You have two options for reading data in Spark:

  • Read directly to the driver and distribute to the executors (not scalable as everything passes through driver)
  • Read directly from the executors

The built in data sources like csv, parquet etc all implement reading from the executors so the job can scale with the data. They define how each partition of the data should be read - e.g. if we have 10 executors, how do you cut up the data source into 10 sections so each executor can directly read one section.

If you want to load from a HTTP request you will either have to read through the driver and distribute, which may be OK if you know the data is going to be less than ~10mb. Otherwise you would need to implement a custom data source to allow the executors to each read partition, can read here for more: https://aamargajbhiye.medium.com/speed-up-apache-spark-job-execution-using-a-custom-data-source-fd791a0fa4b0

Will finish by saying that this second option is almost definitely an anti-pattern. You will likely be much better off providing an intermediate staging environment (e.g. S3/GCS), calling the server to load the data to the intermediate store and then reading to Spark on completion. In scenario 2, you will likely end up putting too much load on the server, amongst other issues.

user_107730880

In previous lifetimes, I created a custom datasource. It is not the most trivial thing to do, but this GitHub repo explains it: https://github.com/jgperrin/net.jgp.books.spark.ch09.

When it comes to reading from a network stream, make sure that only one executor does it.

user_31972440
Related