Same SparkContext across different JVMs

user_16902019 0
Viewed 96

I have a question related to Apache Spark. I work with Java language for writing client code but my question can be answered in any language.

The title of the question may seem like there is already a general question in Google that can be found by a simple search, but the problem is that my question is something else and unfortunately every time I search, I didn't find something about this topic and my requirement. Similar topics that are usually found by searching but not my question is:

  1. Multiple SparkSession for one SparkContext
  2. Multiple SparkSessions in single JVM
  3. ...

My question is not the above questions at all, although it seems similar. I will first explain my question. In the following, after stating the question, I will say my requirement in a higher level because of which I asked the question. My goal is a requirement that will be solved if the question is answered or another solution to the requirement is provided.

The problem I am trying to solve

I wrote a rest server component in which I used Spark Java library. This rest server can receive a series of requests in a specific format and then form a query based on the requests and submit a job through the Spark library functions to the Spark cluster. (My own cluster) Also return the query answer in the form of a asynchronous response (when it is ready and user request it).

I use some code like this to create spark session (summary of it):

SparkConf sparkConf = new SparkConf()
    .setMaster("spark://localhost:7077")
    .setAppName("test");
SparkSession session = SparkSession.builder()
    .config(sparkConf)
    .getOrCreate();
...

As far as I know, we I run above code, spark create application test for me and allocate some of resources from my spark cluster. (I use Spark in standalone mode) for example assume it use all my resources. (So there is no resource for extra new application)

Now I have just one rest server, it can not be scaled at all, and if it goes down, the user can no longer work with the rest server API. So I want to scale it to two instance (at least) on different machines and on different JVMs. (This is the part where my question differs from the others)

If I bring another instance of my rest sever with same code as above, then it will create new Spark session (because it is different JVM on another machine) and it also creates another application with test name in Spark. But since I said all my resources have been used by the first Spark session, this application is on standby and can do nothing. (until resources become free)

Notes about problem:

  1. I do not want to split the cluster resources and add some to the first rest server and some to the second rest server.
  2. I want both versions (or any other numbers of instance if I mentioned) have a single Spark application. In other words, I want same SparkContext across different JVMs. Also note that I submit my spark query as cluster mode in Spark so my application is not worker and one of nodes in cluster becomes driver.

Requirement

As it is clear in the above description, I want my rest server to be HA of type active-active, so that both spark clients are connected to an same application, and the request to the rest servers can be given to each of them. This is my need at a higher level, which may be another way to meet it.

I would be very grateful if there would be a similar application or special documentation or experience, because my searches always ended with questions that I showed at the beginning, while they had nothing to do with my problem. Shame if there is a typo in some parts due to my weakness in English. Thanks.

apache-spark-sql apache-spark java
1 Answers

I like your idea a lot (probably because I had to implement quite a few similar things in the past).

In short, I am 95% sure that there is no way to share JVM, SparkContext between machines, executions, etc. I tried to share dataframes between SparkContext and this was a huge fiasco ;).

The way I would approach that:

  • If your REST server connects to a cluster, once the Spark session is available, register the server to a load balancer.

  • If you submit your REST server as a Spark job, you can have it register to the load balancer.

You can submit multiple job/start multiple server. They can pick any advertise port, which they will share with the load balancer.

Your REST client would interact with the load balancer, not directly with the Spark REST server. Your REST server will have to have healthcheck endpoints so that the load balancer can do its job.

If one of your REST server goes down, the load balancer could start a new one. You will lose the dataframes of your application, but not multiple applications.

If multiple REST servers need to exchange data, I would use Delta as a "cache" or staging zone.

Does that make sense? It should not be too hard to implement and provide a good HA.

user_31972440
Related