What is the difference between using df.as[T] and df.asInstanceOf[Dataset[T]]?

First, asInstanceOf is just telling the compiler to shut up and believe you that df is an instance of the Dataset class (the T part is irrelevant due to type-erasure). In runtime, if the value is not an instance of that class you will get an exception; and in this case it will never be.

On the other hand, as is a method defined in the Dataset class, which asks for an implicit Encoder so it can safely cast the data; note that since the data is processed in runtime, the conversion may still fail.

So the difference is big and you should not use the former.