How to generate a NUL-delimited stream of timestamped filenames with BSD `stat` command

Viewed 181

Let's suppose that you need to generate a NUL-delimited stream of timestamped filenames.

On Linux & Solaris I can do it with:

stat --printf '%.9Y %n\0' -- *

On BSD, I can get the same info, but delimited by newlines, with:

stat -f '%.9Fm %N' -- *

The man talks about a few escape sequences but the NUL byte doesn't seem supported:

If the % is immediately followed by one of n, t, %, or @, then a newline character, a tab character, a percent character, or the current file number is printed.

Is there a way to work around that? edit: (accurately and efficiently?)


Update:

  • Sorry, the glob * is misleading. The arguments can contain any path.

  • I have a working solution that forks a stat call for each path. I want to improve it because of the massive number of files to process.

5 Answers

You may try this work-around solution if running stat command for files:

stat -nf "%.9Fm %N/" * | tr / '\0'

Here:

  • -n: To suppress newlines in stat output
  • Added / as terminator for each entry from stat output
  • tr / '\0': To convert / into NUL byte

Another work-around is to use a control character in stat and use tr to replace it with \0 like this:

stat -nf "%.9Fm %N"$'\1' * | tr '\1' '\0'

This will work with directories also.

Unfortunately, stat out of the box does not offer this option, and so what you ask is not directly achievable.

However, you can easily implement the required functionality in a scripting language like Perl or Python.

#!/usr/bin/env python3

from pathlib import Path
from sys import argv

for arg in argv[1:]:
    print(
        Path(arg).stat().st_mtime,
        arg, end="\0")

Demo: https://ideone.com/vXiSPY

The demo exhibits a small discrepancy in the mtime which does not seem to be a rounding error, but the result could be different on MacOS (the demo platform is Debian Linux, apparently). If you want to force the result to a particular number of decimal places, Python has formatting facilities similar to those of stat and printf.

With any command that can't produce NUL-terminated (or any other character/string terminated) output, you can just wrap it in a function to call the command and then printf it's output with a terminating NUL instead of newline, for example:

nulstat() {
    local fmt=$1 file
    shift
    for file in "$@"; do
        printf '%s\0' "$(stat -f "$fmt" "$file")"
    done
}

nulstat '%.9Fm %N' *

For example:

$ > foo

$ > $'foo\nbar'

$ nulstat '%.9Fm %N' foo* | od -c
0000000    1   6   6   3   1   6   2   5   3   6   .   4   7   7   9   8
0000020    0   1   4   0       f   o   o  \0   1   6   6   3   1   6   2
0000040    5   3   9   .   3   8   8   0   6   9   9   3   0       f   o
0000060    o  \n   b   a   r  \0
0000066

1. What you can do (accurate but slow):

Fork a stat command for each input path:

for p in "$@"
do
    stat -nf '%.9Fm' -- "$p" &&
    printf '\t%s\0' "$p"
done

2. What you can do (accurate but twisted):

In the input paths, replace each occurrence of (possibly overlapping) /././ with a single /./, make stat output /././\n at the end of each record, and use awk to substitute each /././\n by a NUL byte:

#!/bin/bash
shopt -s extglob

stat -nf '%.9Fm%t%N/././%n' -- "${@//\/.\/+(.\/)//./}" |

awk -F '/\\./\\./' '{
    if ( NF == 2 ) {
        printf "%s%c", record $1, 0
        record = ""
    } else
        record = record $1 "\n"
}'

N.B. If you wonder why I chose /././\n as record separator then take a look at Is it "safe" to replace each occurrence of (possibly overlapped) /./ with / in a path?


3. What you should do (accurate & fast):

You can use the following perl one‑liner on almost every UNIX/Linux:

LANG=C perl -MTime::HiRes=stat -e '
    foreach (@ARGV) {
        my @st = stat($_);
        if ( @st > 0 ) {
            printf "%.9f\t%s\0", $st[9], $_;
        } else {
            printf STDERR "stat: %s: %s\n", $_, $!;
        }
    }
' -- "$@"

note: for perl < 5.8.9, remove the -MTime::HiRes=stat from the command line.


ASIDE: There's a bug in BSD's stat:

When %N is at the end of the format string and the filename ends with a newline character, then its trailing newline might get stripped:

For example:

stat -f '%N' -- $'file1\n' file2
file1
file2

For getting the output that one would expect from stat -f '%N' you can use the -n switch and add an explicit %n at the end of the format string:

stat -nf '%N%n' -- $'file1\n' file2
file1

file2

Is there a way to work around that?

If all you need is to just replace all newlines with NULLs, then following tr should suffice

stat -f '%.9Fm %N' * | tr '\n' '\000'

Explanation: 000 is NULL expressed as octal value.

Related