I have a dataframe in Apache Spark with some structs inside the dataframe schema. Now I want to run some Spark SQL query on this dataframe. So far I thought using ` to quote special names inside the dataframe works fine (which it did for names like "foo-bar"). But now I have a field with the name "fääär" and there the parser fails with
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input 'ä' expecting {<EOF>, ',', 'FROM', 'WHERE', 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT', 'SORT', 'CLUSTER', 'DISTRIBUTE'}(line 1, pos 1503)
This is the part of the query it complains about:
SELECT [...] changes.`fäää`.`new` as changes_AL_fäää_new [...]
Someone knows what other quoting I should use here so that it will work? Other SQL statements with this type of quoting work fine (even though it's not required here), e.g.
association.`bla`.`deleted` as association_bla_deleted
If someone wonders why I want to do this: I want to flatten the dataframe to write it to an SQL database.