Problem wih fseek and Microsoft's CRT UNICODE support

user_898348 0
Viewed 97

I'm trying to read a UTF8 encoded text file using the unicode streams. This works fine, but there seems to be a bug with fseek demonstrated with following simple file and program:

The text file I'm reading:

ABC

Raw content of the file

EF BB BF 41 42 43 0D 0A

As you can see the file contains the UTF-8 BOM and the characters ABC followed by an end of line.

The program opens the file using UNICODE support, then it reads one line and displays the raw content of the buffer, which is as expectd. Then it fseeks to the beginning and reads the line again, but this time the content of the buffer is different; there are two bytes at the beginning of the buffer which are actually the BOM for UTF-16 little endian encoded files.

Program

#define _CRT_SECURE_NO_WARNINGS

#include <stdio.h>
#include <stdlib.h>

int main()
{
  FILE *input = _wfopen(L"utf8filewithbom.txt", L"r, ccs=UTF-8");
  if (input == NULL)
  {
    printf("Can't open file\n");
    return 1;
  }

  unsigned char buffer[100];
  fgetws((wchar_t*)buffer, _countof(buffer) / 2, input);
  printf("First 4 bytes of buffer: %02x %02x %02x %02x\n", buffer[0], buffer[1], buffer[2], buffer[3]);

  fseek(input, 0, SEEK_SET);

  fgetws((wchar_t*)buffer, _countof(buffer) / 2, input);
  printf("First 4 bytes of buffer: %02x %02x %02x %02x\n", buffer[0], buffer[1], buffer[2], buffer[3]);

  fclose(input);
}

Expected output:

First 4 bytes of buffer: 41 00 42 00
First 4 bytes of buffer: 41 00 42 00

Actual output:

First 4 bytes of buffer: 41 00 42 00
First 4 bytes of buffer: ff fe 41 00

Is this a bug in the Microsoft CRT or am I doing something wrong?

I'm using Visual Studio 2019 16.4.3.

Things I tried but that didn't change anything:

  • using "rt, ccs=UTF-8" instead of "r, ccs=UTF-8"
  • reading a UTF-16 little endian encoded file instead of an UTF-8 encoded file
utf-8 unicode winapi c
1 Answers

Per the Microsoft docs on fseek:

When the CRT opens a file that begins with a Byte Order Mark (BOM), the file pointer is positioned after the BOM (that is, at the start of the file's actual content). If you have to fseek to the beginning of the file, use ftell to get the initial position and fseek to it rather than to position 0.

Basically, just tweak your code to (comments on changed/added lines):

  FILE *input = _wfopen(L"utf8filewithbom.txt", L"r, ccs=UTF-8");
  if (input == NULL)
  {
    printf("Can't open file\n");
    return 1;
  }
  const long postbomoffset = ftell(input); // Store post-BOM offset

  unsigned char buffer[100];
  fgetws((wchar_t*)buffer, _countof(buffer) / 2, input);
  printf("First 4 bytes of buffer: %02x %02x %02x %02x\n", buffer[0], buffer[1], buffer[2], buffer[3]);

  fseek(input, postbomoffset, SEEK_SET);  // Seek to post-BOM offset, not raw beginning

  fgetws((wchar_t*)buffer, _countof(buffer) / 2, input);
  printf("First 4 bytes of buffer: %02x %02x %02x %02x\n", buffer[0], buffer[1], buffer[2], buffer[3]);

  fclose(input);
user_3646960
Related