DataFrames are dynamically typed, while Datasets and RDDs are statically typed. That means when you define a Dataset or RDD you need to explicitly specify a class that represents the content. This can be useful, because when you go to write transformations on your Dataset, the compiler can check your code for type safety. Take for example this dataset of Pet info. When I use pet.species or pet.name the compiler knows their types at compile time.
case class Pet(name: String, species: String, age: Int, weight: Double)
val data: Dataset[Pet] = Seq(
Pet("spot", "dog", 2, 50.5),
Pet("mittens", "cat", 11, 15.5),
Pet("mickey", "mouse", 1, 1.5)).toDS
println(data.map(x => x.getClass.getSimpleName).first)
// Pet
val newDataset: Dataset[String] = data.map(pet => s"I have a ${pet.species} named ${pet.name}.")
When we switch to using a DataFrame, the schema stays the same and the data is still typed (or structured), but this information is only available at runtime. This is called dynamic typing. This prevents the compiler from catching your mistakes, but it can be very useful because it allows you to write sql like statements and defining new columns on the fly, for example appending columns to an existing DataFrame, without needing to define a new class for every little operation. This flip side is that you can define bad operations that result in nulls or in some cases, runtime errors.
val df: DataFrame = data.toDF
df.printSchema()
// root
// |-- name: string (nullable = true)
// |-- species: string (nullable = true)
// |-- age: integer (nullable = false)
// |-- weight: double (nullable = false)
val newDf: DataFrame = df
.withColumn("some column", ($"age" + $"weight"))
.withColumn("bad column", ($"name" + $"age"))
newDf.show()
// +-------+-------+---+------+-----------+----------+
// | name|species|age|weight|some column|bad column|
// +-------+-------+---+------+-----------+----------+
// | spot| dog| 2| 50.5| 52.5| null|
// |mittens| cat| 11| 15.5| 26.5| null|
// | mickey| mouse| 1| 1.5| 2.5| null|
// +-------+-------+---+------+-----------+----------+
