How can I pretty print a data frame in Hue/Notebook/Scala/Spark?

Question

I am using Spark 2.1 and Scala 2.11 in a HUE 3.12 notebook. I have a dataframe that I can print like this:

df.select("account_id", "auto_pilot").show(2, false)

And the output looks like this:

+--------------------+----------+
|account_id          |auto_pilot|
+--------------------+----------+
|00000000000000000000|null      |
|00000000000000000002|null      |
+--------------------+----------+
only showing top 2 rows

Is there a way of getting the data frame to show as pretty tables (like when I query from Impala or pyspark)?

Impala example of same query:

user_3920656 · Answer

you can use a magic function %table , however this function only works for datasets not dataframe. One option is to convert dataframe to datasets before printing.

import spark.implicits._
case class Account(account_id: String, auto_pilot: String)

val accountDF = df.select("account_id", "auto_pilot").collect()
val accountDS: Dataset[Account] = accountDF.as[Account]

%table accountDS

Right now this is the solution that I can think of. Other better solutions are always welcome. I will modify this as soon I find any other elegant solution.

user_1794744 · Answer

From http://gethue.com/bay-area-bike-share-data-analysis-with-spark-notebook-part-2/

This is what I did

df = sqlContext.sql("select * from my_table")
result = df.limit(5).collect()
%table result

How can I pretty print a data frame in Hue/Notebook/Scala/Spark?

2 Answers