I am writing a filter that reads a file (possibly stdin) and possibly writes to stdout. If the input file starts with a byte order mark I want the output file to have one, otherwise I don't. If I open a file with a BOM as utf-8 then the first character my program reads is the BOM (\uFEFF), which I don't want. If I open a file without a BOM as utf-8-sig it reads it properly, but I can't tell whether to open the output as utf-8 (which will have no signature) or utf-8-sig, which will start the file with a signature mark. So what I want to do is to look at the first character of the file and based on its value decide which encoding to use to open the file. If it were a disk file I could simply close it and reopen it, but because it could be stdin I can't do that. I can work around it by having my program check the first character it reads and output it if it is a signature mark, but I'm wondering if there's a better way. I observe that if I open the file as
f = io.open(inFile, encoding="utf-8", buffering=1)
and then do f.buffer.peek(1) I get back a Unicode object with the first characters of the file. That seems odd because what I would expect from peek() of the bufferd iobase class that has peek() would be a Unicode object of length 1. Now I could look at the first character of what peek returns, and if it is a signature mark read one character and write it to the output, or discard it and then open the output file with utf-8-sig, but I don't really like basing a solution on so much observed, rather than documented, behavior.
Any ideas how to do this right?
[I solved this in my case by having the parsing logic ignore the BOM.]
I also find that when I open the output as
fOut = io.open(sys.stdout.fileno(), mode="at", encoding="utf-8", closefd=False)
I get an extra <CR> at the end of each line (<CR><CR><LF>) which I don't get if I open an actual file.
I tried opening with mode="ab" which didn't work. What does seem to work is to add newline="\n":
fOut = io.open(sys.stdout.fileno(), mode="at", encoding="utf-8", newline="\n", closefd=False)
I think the reason this "works" is because of the same anomaly. As coded it should be using LF as the terminator. In fact, if you feed it input with UNIX line terminators it still outputs DOS line terminators.
I am running Python 2.7.13 on Windows 10.
Thanks.
Gary