What is the equivalent of utf8-sig in pyspark?

Viewed 373

I'm trying to find the equivalent of utf-8-sig in pyspark.

If my file contains headers, it cannot be read anymore with utf-8 (i don't know why).

I've tried utf-8-sig .option("encoding", "utf-8-sig") but get the following error :

java.nio.charset.UnsupportedCharsetException: utf-8-sig

It works with utf-16, but sometimes the file contains headers, sometimes not. So i would prefer a dynamic solution.

Is there any equivalent to utf-8-sig?

0 Answers
Related