Why does reading a file to a std::string have a wrong length on Windows?

user_3764804 0
Viewed 161

I've tried various idiomatic C++ solutions for reading a file into a std::string on Windows and it has been giving me tons of headaches since the std::string length was consistently wrong (e.g. much smaller than the original file). The same codes all worked perfectly on Linux though.

The following functions all misbehave on Windows:

#include <boost/filesystem.hpp>
#include <iostream>
#include <sstream>

using namespace boost::filesystem;
using namespace boost::iostreams;

std::string read_string_from_file_using_streams(const path &file_path) {
    return read_string_from_file_using_streams(file_path.parent_path(), file_path.filename().string());
}

std::string read_string_from_file_using_streams(const path &parent_directory, const std::string &file_name) {
    const auto original_file_path =
            parent_directory.string() + (char) path::preferred_separator + file_name;
    const std::ifstream input_stream(original_file_path);

    if (input_stream.fail()) {
        throw std::runtime_error("File is not readable");
    }

    std::stringstream buffer;
    buffer << input_stream.rdbuf(); // Does not read the whole file on Windows!

    return buffer.str();
}

std::string read_string_from_file_with_string_pre_allocation(const std::string& path) {
    std::ifstream t(path);
    std::string str;

    t.seekg(0, std::ios::end);
    str.reserve(t.tellg()); // Reserves the correct length
    t.seekg(0, std::ios::beg);

    str.assign(std::istreambuf_iterator<char>(t), // Does not read the whole file on Windows!
        std::istreambuf_iterator<char>());

    return str;
}

std::string read_string_from_file_using_if_stream(const std::string& path) {
    std::ifstream file(path);
    return std::string(std::istreambuf_iterator<char>(file), std::istreambuf_iterator<char>()); // Does not read the whole file on Windows!
}

Finally, the following C standard code example works properly:

std::string read_using_c_standard_library(const std::string& path) {
    auto f = fopen(path.c_str(), "rb");
    fseek(f, 0, SEEK_END);
    long fsize = ftell(f);
    fseek(f, 0, SEEK_SET);

    auto string = (char *) malloc(fsize + 1);
    fread(string, 1, fsize, f); // This reads the whole file on Windows, finally!
    fclose(f);

    string[fsize] = 0;

    std::string s(string, fsize);
    free(string);
    return s;
}

I don't understand why this is happening. What's wrong with the C++ code approaches? My example file is 516 KB and the buggy functions only read e.g. 912 bytes.

file-io windows c++
2 Answers

Can you see the difference?

const std::ifstream input_stream(original_file_path);

auto f = fopen(path.c_str(), "rb");

In the second case the file is opened in binary mode, in the first it isn't.

That said the code should still read the whole file, but it will change the line endings so the size of the resulting string might not be what you expect.

user_8820030

On Windows, the line endings of text files are normally \r\n, on Linux they are just \n.

Opening a file in text mode causes the \r\n line endings to be converted to \n, so that they are \n on both platforms.

However, this should only be done for text files, not for binary files, because otherwise all \r\n byte sequences will be converted to \n, thereby corrupting the data and making the file shorter.

Also, in text mode on Windows, a byte with the value 0x1A is interpreted as the end of file (EOF) marker, even if the actual file length is longer. In binary mode though, a byte with this value is not interpreted to have any special meaning.

The line

std::ifstream file(path);

will open the file in text mode, whereas the line

auto f = fopen(path.c_str(), "rb");

will open the file in binary mode.

This is probably the reason why you are getting different file lengths.

user_121494710
Related