sql query translate to spark lang in zeppelin

user_12225304 0
Viewed 63

I try to tranfer sql query to spark lang. Here is the sql query:

%sql
select ${checkbox:fields=AGE|NAME|SEX} from table limit ${limit=10}

Then I transfer spark lang:

%spark
var fields = z.select("fields", Seq(("AGE","AGE"),("NAME","NAME"),("SEX","SEX"))
val lim = z.input("limit","20")
df.selectExpr(fields).limit(lim).show()

but it does not work. How to collect it?

apache-zeppelin apache-spark scala sql
1 Answers

There are couple of things that you have missed here.

  • In sql you are using checkbox while in scala/spark you are using select. In checkbox you can select multiple values while in select you can select only 1 value. In rest of answer I am assuming you want to do multiple selection.

  • Try this in scala:

 val f = z.checkbox("fields", options).to[collection.immutable.Seq]
 val lim = z.input("limit","20").asInstanceOf[String].toInt
 //Assuming df is already defined
 df.selectExpr(f.map(c => (c+"")): _*).limit(lim).show()
 df.select(f.map(c => col(c+"")): _*).limit(lim).show()

z.input returns an object. limit in spark DF takes Int as input, so we have converted it to Int. z.checkbox returns scala.collection.Iterable, while selectExpr/select needs the input in different types.

user_24400190
Related