Byte order mark screws up file reading in Java

Here is my code to read csv files in most char sets. It should cover 99% situations.

        try(InputStream inputStream = new FileInputStream(csvFile);){
            BOMInputStream bomInputStream = new BOMInputStream(inputStream ,ByteOrderMark.UTF_8, ByteOrderMark.UTF_16LE, ByteOrderMark.UTF_16BE, ByteOrderMark.UTF_32LE, ByteOrderMark.UTF_32BE);
            Charset charset;
            if(!bomInputStream.hasBOM()) charset = StandardCharsets.UTF_8;
            else if(bomInputStream.hasBOM(ByteOrderMark.UTF_8)) charset = StandardCharsets.UTF_8;
            else if(bomInputStream.hasBOM(ByteOrderMark.UTF_16LE)) charset = StandardCharsets.UTF_16LE;
            else if(bomInputStream.hasBOM(ByteOrderMark.UTF_16BE)) charset = StandardCharsets.UTF_16BE;
            else { throw new Exception("The charset of the file " + csvFile + " is not supported.");}
            
            try(Reader streamReader = new InputStreamReader(bomInputStream, charset);
                BufferedReader bufferedReader = new BufferedReader(streamReader);) {
                for(String line; (line = bufferedReader.readLine()) != null; ) {
                    String[] columns = line.split(",");
             //read csv columns
            }
        }

IMO none of the given answers is really satisfying. Just skipping the BOM and then read the rest of the stream in the current platform's default encoding is definitively wrong. Remember: The platform default on Unix/Linux and windows differ: former is UTF-8, later is ANSI. Such a solution only works if the rest of the stream (after the BOM) only contains 7-bit ASCII characters (which, I admit, in most programmer near files like configurations is true). But as soon there are non ASCII characters, you will fail with this approach.

That's why all java classes/methods, which can convert byte arrays/streams to string (and vice versa) have a second parameter indicating the encoding to be used (Reader, Writer, Scanner, String.getBytes(), etc.).

There are so much character encodings out in the world, not only UTF-xx. And still - in the current year 2021 - there are so much encoding problems between end user applications, especially if they run on different platforms (iOS, windows, unix). All these problems only exist because the programmer was too lazy to learn how character encoding works.

Thus, it's an absolute MUST to evaluate first the encoding to be used, and then performing the string/stream conversion using the found encoding. Consulting the respective specification(s) is the first step. And only if you cannot be sure which encoding you encounter while reading a stream you have to evaluate it by yourself. But caution: such an evaluation always will only be a 'best guess', there is no algorithm which can cover all possibilities.

In this sense, Lee's answer (and coding example) from Feb 6,2021 is IMO the best one, except that he falls back to UTF-8 if there is no BOM.

I had the same problem, and because I wasn't reading in a bunch of files I did a simpler solution. I think my encoding was UTF-8 because when I printed out the offending character with the help of this page: Get unicode value of a character I found that it was \ufeff. I used the code System.out.println( "\\u" + Integer.toHexString(str.charAt(0) | 0x10000).substring(1) ); to print out the offending unicode value.

Once I had the offending unicode value, I replaced it in the first line of my file before I went on reading. The business logic of that section:

String str = reader.readLine().trim();
str = str.replace("\ufeff", "");

This fixed my problem. Then I was able to go on processing the file with no issue. I added on trim() just in case of leading or trailing whitespace, you can do that or not, based on what your specific needs are.

NotePad++ is a good tool to convert UTF-8 encoding to UTF-8(BOM) encoding.

https://notepad-plus-plus.org/downloads/

UTF8BOMTester.java

public class UTF8BOMTester {

public static void main(String[] args) throws FileNotFoundException, IOException {
    // TODO Auto-generated method stub
    File file = new File("test.txt");
    boolean same = UTF8BOMInputStream.isSameEncodingType(file);
    System.out.println(same);
    if (same) {
        UTF8BOMInputStream is = new UTF8BOMInputStream(file);
        BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
        System.out.println(br.readLine());
    }

}

static void bytesPrint(byte[] b) {
    for (byte a : b)
        System.out.printf("%x ", a);
}}

UTF8BOMInputStream.java

public class UTF8BOMInputStream extends InputStream {

byte[] SYMBLE_BOM = { (byte) 0xEF, (byte) 0xBB, (byte) 0xBF };
FileInputStream fis;
final boolean isSameEncodingType;
public UTF8BOMInputStream(File file) throws IOException {
    FileInputStream fis=new FileInputStream(file);
    byte[] symble=new byte[3];
    fis.read(symble);
    bytesPrint(symble);
    isSameEncodingType=isSameEncodingType(symble);
    if(isSameEncodingType)
        this.fis=fis;
    else
        this.fis=null;
    
}

@Override
public int read() throws IOException {
    return fis.read();
}

void bytesPrint(byte[] b) {
    for (byte a : b)
        System.out.printf("%x ", a);
}

boolean bytesCompare(byte[] a, byte[] b) {
    if (a.length != b.length)
        return false;

    for (int i = 0; i < a.length; i++) {
        if (a[i] != b[i])
            return false;
    }
    return true;
}
boolean isSameEncodingType(byte[] symble) {
    return bytesCompare(symble,SYMBLE_BOM);
}
public static boolean isSameEncodingType(File file) throws IOException {
    return (new UTF8BOMInputStream(file)).isSameEncodingType;
}