I'm trying to find the equivalent of utf-8-sig in pyspark.
If my file contains headers, it cannot be read anymore with utf-8 (i don't know why).
I've tried utf-8-sig .option("encoding", "utf-8-sig") but get the following error :
java.nio.charset.UnsupportedCharsetException: utf-8-sig
It works with utf-16, but sometimes the file contains headers, sometimes not. So i would prefer a dynamic solution.
Is there any equivalent to utf-8-sig?