Apache Spark shortest job scala

Question

I am new to Apache Spark and scala programming. I am writing a code in scala using apache spark api docs. My goal is to create a graph and deploy objects and compute shortest path.I have written a program to generate a csv file of object which I want to use. It consists of vehicleID,source,Destination. It is as follows:

[My sample csv file][1]


  [1]: https://i.stack.imgur.com/KtSVz.png

My code to generate CSV file

import java.io.BufferedWriter
import java.io.FileWriter

import scala.collection.JavaConverters._
import scala.collection.mutable.ListBuffer
import scala.util.Random
import au.com.bytecode.opencsv.CSVWriter

import scala.collection.mutable

class MakeCSV() {
  def csvBuilder(dx:Int){
    val outputfile= new BufferedWriter(new FileWriter("vehicles.csv"))
    val csvWriter= new CSVWriter(outputfile)
    val csvFields= Array("Vehicle-id","Source","Destination")
    val vehicleID=(0 to dx).toList
    val sourceList=mutable.MutableList[String]()
    val destinationList=mutable.MutableList[String]()
    var i,sx,sy,dsx,dsy=0
    for(i<-0 to dx){
      sx=Random.nextInt(dx)
      sy=Random.nextInt(dx)
      dsx=Random.nextInt(dx)
      dsy=Random.nextInt(dx)
      sourceList.+=((sx,sy).toString())
      destinationList.+=((dsx,dsy).toString())
    }
    var listOfRecords = new ListBuffer[Array[String]]()
    listOfRecords += csvFields
    for (i<- 0 to dx){
      listOfRecords+=Array(i.toString,sourceList(Random.nextInt(sourceList.length)),destinationList(Random.nextInt(destinationList.length)))
    }
    csvWriter.writeAll(listOfRecords.asJava)
    csvWriter.close()
  }
}

My main file:

import java.io.PrintWriter

import scala.io.StdIn
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.graphx.Graph
import org.apache.spark.graphx.util.GraphGenerators
object MainFile {
  def main(args:Array[String]):Unit={

//    Vehicle CSV file Generation
    println("Enter the number of cars")
    val input=StdIn.readInt()
    val makecsv= new MakeCSV()
    makecsv.csvBuilder(input)

//   Spark Job Configuration
    val conf = new SparkConf().setAppName("DjikstraShortestPath")
    val sc= new SparkContext(conf)

//    Graph Generation
    println("Enter the number of rows for grid")
    val row= StdIn.readInt()
    println("Enter the number of columns for grid")
    val column = StdIn.readInt()
    val graph:Graph[(Int, Int), Double]=GraphGenerators.gridGraph(sc,row,column)

//    Vehicle File opening
//    For each Vehicle compute shortest path using source destination in csv file

  }
}

Now I want to open that csv file and using its source and destination I want to compute shortest path for each vehicle using the graph generated above. Can anyone help me? How to open the csv file read it and find shortest path

Apache Spark shortest job scala

0 Answers