I'm new to Spark and Scala.
We have an external data source feeding us JSON. This JSON has quotes around all values including number and boolean fields. So by the time I get it into my DataFrame all the columns are strings. The end goal is to convert these JSON records into a properly typed Parquet files.
There are approximately 100 fields, and I need to change several of the types from string to int, boolean, or bigint (long). Further, each DataFrame we process will only have a subset of these fields, not all of them. So I need to be able to handle subsets of columns for a given DataFrame, compare each column to a known list of column types, and cast certain columns from string to int, bigint, and boolean depending on which columns appear in the DataFrame.
Finally, I need the list of column types to be configurable because we'll have new columns in the future and may want to get rid of or change old ones.
So, here's what I have so far:
// first I convert to all lower case for column names
val df = dfIn.toDF(dfIn.columns map(_.toLowerCase): _*)
// Big mapping to change types
// TODO how would I make this configurable?
// I'd like to drive this list from an external config file.
val dfOut = df.select(
df.columns.map {
///// Boolean
case a @ "a" => df(a).cast(BooleanType).as(a)
case b @ "b" => df(b).cast(BooleanType).as(b)
///// Integer
case i @ "i" => df(i).cast(IntegerType).as(i)
case j @ "j" => df(j).cast(IntegerType).as(j)
// Bigint to Double
case x @ "x" => df(x).cast(DoubleType).as(x)
case y @ "y" => df(y).cast(DoubleType).as(y)
case other => df(other)
}: _*
)
Is this a good efficient way to transform this data to having the types I want in Scala?
I could use some advice on how to drive this off an external 'config' file where I could define the column types.