Slow performance of C++ std::filesystem::directory_iterator for symlinks

Viewed 1101

I understand std::filesystem is not a speed demon. But compared to opendir/readdir I see execution times for std::filesystem::directory_iterator being a factor 8-10 larger (Ubuntu 18.04, clang 9.0.0). Now I wonder if I do something wrong or at least hope for some insight why it is (so much) slower.

In more detail: I'm comparing these two, almost identical functions:

    int32_t GetNumberOfFilesStdFilesys(const std::filesystem::path& testDir) {
    // Find all files in that testDir
    int32_t numFiles = 0;
    for (auto& dirent : std::filesystem::directory_iterator(testDir)) {
        if (dirent.is_regular_file() || dirent.is_symlink()) {
            // const std::string filename(dirent.path().filename());
            numFiles++;
        }
    }
    return numFiles;
}

int32_t GetNumberOfFilesStdC(const std::filesystem::path& testDir) {
    // Find all files in that testDir
    int32_t numFiles = 0;
    DIR* dir = opendir(testDir.string().c_str());
    for (struct dirent* entry = readdir(dir); entry != NULL; entry = readdir(dir
)) {
        if (entry->d_type == DT_REG || entry->d_type == DT_LNK) {
            // const std::string filename(entry->d_name);
            numFiles++;
        }
    }
    closedir(dir);
    return numFiles;
}

They walk over a directory and count the regular and symbolic-link entries. I then run this in a loop 30000 times for a directory with 100 symlinks and 300 times for a directory with 10000 symlinks. Here the result:

Symlink (Number Files: 100, Iterations: 30000)
    Std C lib       CPU time: 443.91 msec       Wall clock: 446.97 msec
    std::filesystem     CPU time: 3908.20 msec      Wall clock: 3910.18 msec
Symlink (Number Files: 10000, Iterations: 300)
    Std C lib       CPU time: 455.26 msec       Wall clock: 455.92 msec
    std::filesystem     CPU time: 5090.57 msec      Wall clock: 5090.61 msec

the CPU time is measured with std::clock and the Walltime with std::chrono::high_resolution_clock. I assume that directory_iterator is processing more data, maybe following the symlink (?) - are there any flags to prevent this? Or is this simply the price of platform abstraction?

Thanks for any insight!

0 Answers
Related