Ruby: Check for Byte Order Marker

Question

In Rails, we are some text files as ISO-8859-1. Sometimes the files come in as UTF-8 with BOM. I am trying to determine if its UTF-8 with BMO then re-read the file as bom|UTF-8.

I trying the following but it doesn't seem to compare correctly:

# file is saved as UTF-8 with BOM using Sublime Text 2

> string = File.read(file, encoding: 'ISO-8859-1')

# this doesn't work, while it supposed to work
> string.start_with?("\xef\xbb\xbf".force_encoding("UTF-8"))
> false

# it works if I try this
> string.start_with?('ï»¿')
> true

The purpose is to read the file as UTF-8 with BOM if file has the Byte Order Marker at the start and I want to avoid string.start_with?('ï»¿') method.

user_1182464 · Answer

This didn't work for me, I had to check the bytes.

string[0].bytes ==  [239, 187, 191] # true for UTF-8 + BOM

See BOM for other encodings

If you just want to check the file, and reopen it properly (e.g. File.open(file, "r:bom|utf-8")).

Then you don't need the whole file, just read the 3 first bytes

is_bom = File.open(file) { |f| f.read(3).bytes ==  [239, 187, 191] }

Ruby: Check for Byte Order Marker

2 Answers