Fastest way to create 5,000,000,000 empty files in 5,000 directories in bash

user_14276818 0
Viewed 747

So I need to create 5,000,000,000 files on a linux drive - empty files. What is the fastest / most efficient way to achieve this?

What I am currently trying:

for dir in /xfs/*/; do seq 1 1000000 | xargs -n1 -I% bash -c 'touch '$dir/file%' ; done;

This is "working," albeit very slowly. (creating 1,000,000 files in 5,000 directories).

Is there a better way to accomplish this? (without running in to the Argument list too long issue)

TIA

file-io file loops bash
3 Answers

touch is an external program that you are having to start 5,000,000,000 times. You are also running seq 5,000 times. Don't run either of them.

for dir in /xfs/*/; do
    for ((i=1; i <= 1000000; i++)); do
        : > "$dir/file$i"
    done
done

This loop is a single shell command that spawns no additional processes; the : command's output redirection is handled by a single system call in the current shell.

user_11268410

The biggest problem with your code is xargs -n1 -I% bash -c 'touch. For each file to be created you are executing two processes - first bash, then touch. Spawning processes takes time and two for a single file is going to be a lot.

The following could be faster, but I suspect the pure bash solution in another answer may be faster anyway. find outputs a list of directories inside /xfs/. Then seq outputs that directory with the filename suffixed with a number. Then the resulting strings are passed to touch.

find /xfs/ -mindepth 1 -maxdepth 1 -type d |
xargs -d'\n' -I{} seq -f "{}/file%.0f" 1000000 |
xargs -d'\n' touch

If you have GNU xargs, you could easily parallelize the execution by adding -P<num> to the last xargs.

user_90727530

Split nothing to millions. This is fast because it is one process, that immediately decides what the output is, and just create empty files.

for dir in ./*/; do
    split -d -n1000000 /dev/null "$dir/file"
done

I noticed similar times for this modification:

find . -type d | xargs -I{} split -d -n1000000 /dev/null {}/file
user_75896360
Related