I'm testing a case where whatever columns are included in Join are actually present in the correct dataframe or not.
Let's say i have 2 DF's defined as a case class
case class JoinResult(
id: Option[String],
name: Option[String],
maxAttack: Option[Int],
maxDefense: Option[Int],
cartoon: Option[String]
)
case class JoinSchema(
uuid: String,
cartoon: String
)
In this JoinResult schema is the parentDF and JoinSchema is childDF
To join this i'm running a spark job which will take a dslString Something like
"""
|{
| "with": "D2",
| "type": "left",
| "keys": {
| "D1.id": "D2.uuid"
| },
| "colList": [
| {
| "col": "D1.id",
| "as": "id"
| },
| {
| "col": "D1.name",
| "as": "name"
| },
| {
| "col": "D1.max_attack",
| "as": "max_attack"
| },
| {
| "col": "D1.max_defense",
| "as": "max_defense"
| },
| {
| "col": "D2.cartoon",
| "as": "cartoon"
| },
| {
| "col": "D2.test_cols",
| "as": "test_cols"
| }
| ]
|}
|""".stripMargin
My End requirement is to check all the cols which start with D2 aliasing should be there in childDF and cols starting with D1 aliasing should be there in parentDF And if none of the column is there then it should throw me an error.
To achieve this i'm writing a function to iterate over the col present in the dslString and the ones starting with D1 should be stored into parentInvCols and cols starting with D2 should be stored in childInvCols.
Here is the main function
config.colList.foreach { cInfo =>
val colInfo = cInfo.col.split("\\.")
if (childAlias.equals(colInfo.head)) {
colInfo.last
}
}
So as per the code if childAlias = colInfo.head means D2 = D2 then it should store in childInvCols