In the documentation, I see mention of user-defined functions: https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html
But this is showing Java and Scala examples.
I am talking about Spark SQL code like the one below. Is it possible to define and use UDFs for that?
%sql
CREATE TABLE bla USING PARQUET AS
SELECT *
FROM jaja.haha AS haha
I am confused why the guide is saying SQL Reference and then proceeds to show the non-SQL code.
Starting with Databricks Runtime 9.0, it's possible to define a scalar user-define function using the SQL only syntax (example from release notes):
CREATE FUNCTION square(x DOUBLE) RETURNS DOUBLE RETURN x * x;
Consult documentation for syntax & more examples.
You can do that using CREATE FUNCTION.
CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [ IF NOT EXISTS ]
function_name AS class_name [ resource_locations ]
Read databricks article here.