Unsure how to avoid the following type error: found: Any required: String

user_13136602 0
Viewed 228

This is what my data looks like:

configStr: String =
"
{
  "validation": {
    "target_feed": "tables.validation",
    "data_validations":
        [
            {"program": "program1",
            "test_description": "Checking if column1 are distinct",
            "input_column": "column1",
            "test": "distinctness",
            "query": "select * from table1",
            "condition": "None"},
            {"program": "program12",
            "test_description": "Checking if column2 are distinct",
            "input_column": "column2",
            "test": "Anomaly",
            "query": "select * from table2",
            "condition": "None"}
        ]
  }
}"

I need to iterate through the data validations and leverage each of those fields. I'm planning to do that like this:

val resultsAsDf = conf("test")
.asInstanceOf[Map[String, Any]]("data_validations")
.asInstanceOf[Seq[Map[String, Any]]]
.map{ dv => someFunc(dv) }
.reduce(_.unionAll(_))

Now, to create someFunc that will handle the logic. I build something like this:

def someFunc(testCase: Map[String, Any]): Unit = {

    if (testCase("test") == "distinctness") {
        
        val tempDF = spark.sql(testCase("query"))
        
        val verificationResults: VerificationResult = { VerificationSuite()
        .onData(tempDF)                                                    
        .addCheck(
        Check(CheckLevel.Error, testCase("program"))
        .hasDistinctness(testCase("column"), Check.IsOne))
        .run()
        }                                                           
    }
    else{
        println("Nothing")
    }
}

Now, I get the following errors:

<console>:54: error: type mismatch;
 found   : Any
 required: String
               val tempDF = spark.sql(testCase("query"))
                                              ^
<console>:60: error: type mismatch;
 found   : Any
 required: String
               Check(CheckLevel.Error, testCase("program"))

The thing is, I need the mapped values to be various types which is why I chose Any. Is there any way around this or am I doing something wrong?

apache-spark scala
2 Answers

SequenceToSequence, who is almost correct by the way, mixed up which type needs to change. testCase is returning Any, when spark.sql(_) expects a String, the problem isn't the return of the someFunc it's the return of the Map.

def someFunc(testCase: Map[String, Any]): Unit = {}

val tempDF = spark.sql(testCase("query"))

// Spark 3.1.1 ScalaDoc
def sql(sqlText: String): DataFrame

So when you pass your String key into the map, you get an Any, which is not a String as spark.sql expects:

testCase: Map[String, Any]

testCase("string") // Result Any

Reference documentation: SparkSession ScalaDoc

user_50654000

Your someFunc returns Unit, which is similar to java's 'void' type, however, spark.sql requires a String. You will need to modify your testCase to return a string that contains a query.

def someFunc(testCase: Map[String, Any]): String = { ... }
user_30509860
Related