Preallocate disk space for a file in Python without changing its size

Viewed 667

I'm writing a program that downloads several files at once from several different servers (one download thread per server, of course!). I'm worried about having multiple files growing on disk simultaneously causing disk fragmentation and I'd like to mitigate that by preallocating space on disk for the full file's length (as reported by the Content-Length header) before starting the download, ideally without increasing the file's apparent length (so I can resume failed downloads just by opening the partially downloaded file in append mode).

Is that possible in a platform-independent manner?

2 Answers

I did a bit of googling and found this lovely article with some C code to do exactly what you're asking on Windows. Here's that C code translated to ctypes (written for readability):

    import ctypes
    import msvcrt
    # https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setfileinformationbyhandle
    set_file_information = ctypes.windll.kernel32.SetFileInformationByHandle

    class AllocationInfo(ctypes.Structure):
        _fields_ = [('AllocationSize', ctypes.c_longlong)]
    
    def allocate(file, length):
        """Tell the filesystem to preallocate `length` bytes on disk for the specified `file` without increasing the
        file's length.
        In other words, advise the filesystem that you intend to write at least `length` bytes to the file.
        """
        allocation_info = AllocationInfo(length)
        retval = set_file_information(ctypes.c_long(msvcrt.get_osfhandle(file.fileno())),
                                      ctypes.c_long(5),  # constant for FileAllocationInfo in the FILE_INFO_BY_HANDLE_CLASS enum
                                      ctypes.pointer(allocation_info),
                                      ctypes.sizeof(allocation_info)
                                      )
        if retval != 1:
            raise OSError('SetFileInformationByHandle failed')

This will change the file's Size on disk: as shown in file explorer to the length you specify (plus a few kilobytes for metadata), but leave the Size: unchanged.

However, in the half hour I've spent googling, I've not found a way to do that on POSIX. fallocate() actually does the exact opposite of what you're after: it sets the file's apparent length to the length you give it, but allocates it as a sparse extent on the disk, so writing to multiple files simultaneously will still result in fragmentation. Ironic, isn't it, that Windows has a file management feature that POSIX lacks?

I'd love nothing more than to be proven wrong, but I don't think it's possible.

FILENAME = "somefile.bin"
SIZE = 4200000

with open(FILENAME, "wb") as file:
    file.seek(SIZE - 1)
    file.write(b"\0")

Advantages:

  1. Portable across all platforms.
  2. Very efficient if you'd be mmaping (memory-mapping) the files to perform writes on them (via MADV_SEQUENTIAL if sequential access is needed).
Related