Node.js delete first N bytes from a file

user_926841 0
Viewed 585

How to delete (remove | trim) N bytes from the beginning of a binary file without loading it in the memory?

We have fs.ftruncate(fd, len, callback), which cuts out bytes from the end of the file (if it is bigger).

How to cut bytes from the beginning, or trim from the beginning in Node.js without reading a file in the memory?

I need something like truncateFromBeggining(fd, len, callback) or removeBytes(fd, 0, N, callback).

If it is not possible, what is the fastest way to do it with file streams?

On most filesystems you can't "cut" a part out from the beginning or from the middle of a file, you can only truncate it at the end.

Having the above in mind I imagine, we have to probably open the input file stream, to seek to after the Nth byte, and to pipe the rest of the bytes to an output file stream.

node.js truncate filestream file
2 Answers

You're asking for an OS file system operation: the ability to remove some bytes from the beginning of a file in place, without rewriting the file.

You're asking for a file system operation that does not exist, at least in Linux / FreeBSD / MacOS / Windows.

If your program is the only user of the file and it fits in RAM, your best bet is to read the whole thing into RAM, then reopen the file for writing, then write out the part you want to keep.

Or you can create a new file. Let's say your input file is called q. Then you'd create a file called, maybe new_q with a stream attached. You'd pipe the contents you wanted to the new file. Then you'd unlink (delete) the input file q and rename the output file new_q to q.

Careful: this unlink / rename operation will create a short time when no file named q is available. So if some other program tries to open it and doesn't find it, it should try again a few times.

If you're creating a queueing scheme, you might consider using some other scheme to hold your queue data. This file read / rewrite / unlink / rename sequence has lots of ways it can go wrong on you under heavy load. (Ask me how I know that when you have a couple of hours to spare ;-) redis is worth a look.

user_2056080

I decided to solve the problem in bash.

The script truncates the files in a temp folder first, then moves them back to the original folder.

The truncate is done with tail:

tail --bytes="$max_size" "$from_file" > "$to_file"

The full script:

#!/bin/bash

declare -r store="/my/data/store"
declare -r temp="/my/data/temp"
declare -r max_size=$(( 200000 * 24 ))

or_exit() {
    local exit_status=$?
    local message=$*

    if [ $exit_status -gt 0 ]
    then
        echo "$(date '+%F %T') [$(basename "$0" .sh)] [ERROR] $message" >&2
        exit $exit_status
    fi
}

# Checks if there are any files in 'temp'. It should be empty.
! ls "$temp/"* &> '/dev/null'
    or_exit 'Temp folder is not empty'

# Loops over all the files in 'store'
for file_path in "$store/"*
do
    # Trim bigger then 'max_size' files from 'store' to 'temp'
    if [ "$( stat --format=%s "$file_path" )" -gt "$max_size" ]
    then
        # Truncates the file to the temp folder
        tail --bytes="$max_size" "$file_path" > "$temp/$(basename "$file_path")"
            or_exit "Cannot tail: $file_path"
    fi
done
unset -v file_path

# If there are files in 'temp', move all of them back to 'store'
if ls "$temp/"* &> '/dev/null'
then
    # Moves all the truncated files back to the store
    mv "$temp/"* "$store/"
        or_exit 'Cannot move files from temp to store'
fi
user_9268410
Related