Originally, my project involved batch data preparation using Spark. Now it's being ported to real-time using Spring Boot.
- Spring boot consumes a JSON request and parses data to Bean class
- Data prep is performed
- Machine learning is performed on prepped data
- Response is sent
I'm running the data prep part as in the following example. Note that it's important that the data prep code (queries originally written in a notebook) is preserved for maintainability.
Account account = data.Account;
Encoder<Account> accountEncoder = Encoders.bean(Account.class);
Dataset<Account> accountDS = spark.createDataset(
Collections.singletonList(account), accountEncoder);
accountDS.createOrReplaceTempView("account");
Dataset<Row> accountQuery = spark.sql("select * from account where....); // complex query here
In short, the Bean object is parsed by Spark and a data frame is created. This code recreates the tables in a way that replicates the database environment.
This is done for multiple objects/tables, and later queries refer temp views created earlier. Needless to say this takes forever and it's not compatible with a real time application.
Does Spark support any way to make these computations faster? For example, pre-caching the schemas and the query execution plans and then just recomputing the results as new data arrives?