Can i cache the output of a command on Linux from CLI?

user_362420 0
Viewed 6355

I'm looking for an implementation of a 'cacheme' command, which 'memoizes' the output of whatever has in ARGV. If it never ran it, it will run it and somewhat memorize the output. If it ran it, it will just copy the output of the file (or even better, both output and error to &1 and &2 respectively).

Let's suppose someone wrote this command, it would work like this.

$ time cacheme sleep 1    # first time it takes one sec
real   0m1.228s
user   0m0.140s
sys    0m0.040s

$ time cacheme sleep 1    # second time it looks for stdout in the cache (dflt expires in 1h)
#DEBUG# Cache version found! (1 minute old)

real   0m0.100s
user   0m0.100s
sys    0m0.040s

This example is a bit silly because it has no output. Ideally it would be tested on a script like sleep-1-and-echo-hello-world.sh.

I created a small script that creates a file in /tmp/ with hash of full command name and username, but I'm pretty sure something already exists.

Are you aware of any of this?

command-line-interface caching command-line bash
8 Answers

Author of bash-cache here with an update. I recently published bkt, a CLI and Rust library for subprocess caching. Here's a simple example:

# Execute and cache an invocation of 'date +%s.%N'
$ bkt -- date +%s.%N
1631992417.080884000

# A subsequent invocation reuses the same cached output
$ bkt -- date +%s.%N
1631992417.080884000

It supports a number of features such as asynchronous refreshing (--stale and --warm), namespaced caches (--scope), and optionally keying off the working directory (--cwd) and select environment variables (--env). See the README for more.

It's still a work in progress but it's functional and effective! I'm using it already to speed up my shell prompt and a number of other common tasks.

user_1136320

I created bash-cache, a memoization library for Bash, which works exactly how you're describing. It's designed specifically to cache Bash functions, but obviously you can wrap calls to other commands in functions.

It handles a number of edge-case behaviors that many simpler caching mechanisms miss. It reports the exit code of the original call, keeps stdout and stderr separately, and retains any trailing whitespace in the output ($() command substitutions will truncate trailing whitespace).

Demo:

# Define function normally, then decorate it with bc::cache
$ maybe_sleep() {
  sleep "$@"
  echo "Did I sleep?"
} && bc::cache maybe_sleep

# Initial call invokes the function
$ time maybe_sleep 1
Did I sleep?

real    0m1.047s
user    0m0.000s
sys     0m0.020s

# Subsequent call uses the cache
$ time maybe_sleep 1
Did I sleep?

real    0m0.044s
user    0m0.000s
sys     0m0.010s

# Invocations with different arguments are cached separately
$ time maybe_sleep 2
Did I sleep?

real    0m2.049s
user    0m0.000s
sys     0m0.020s

There's also a benchmark function that shows the overhead of the caching:

$ bc::benchmark maybe_sleep 1
Original:       1.007
Cold Cache:     1.052
Warm Cache:     0.044

So you can see the read/write overhead (on my machine, which uses tmpfs) is roughly 1/20th of a second. This benchmark utility can help you decide whether it's worth caching a particular call or not.

user_1136320

Improved upon solution from error:

  • Pipes output into the "tee" command which allows it to be viewed real-time as well as stored in the cache.
  • Preserve colors (for example in commands like "ls --color") by using "script --flush --quiet /dev/null --command $CMD".
  • Avoid calling "exec" by using script as well
  • Use bash and [[
    #!/usr/bin/env bash

    CMD="$@"
    [[ -z $CMD ]] && echo "usage: EXPIRY=600 cache cmd arg1 ... argN" && exit 1

    # set -e -x

    VERBOSE=false
    PROG="$(basename $0)"

    EXPIRY=${EXPIRY:-600}  # default to 10 minutes, can be overriden
    EXPIRE_DATE=$(date -Is -d "-$EXPIRY seconds")

    [[ $VERBOSE = true ]] && echo "Using expiration $EXPIRY seconds"

    HASH=$(echo "$CMD" | md5sum | awk '{print $1}')
    CACHEDIR="${HOME}/.cache/${PROG}"
    mkdir -p "${CACHEDIR}"
    CACHEFILE="$CACHEDIR/$HASH"

    if [[ -e $CACHEFILE ]] && [[ $(date -Is -r "$CACHEFILE") > $EXPIRE_DATE ]]; then
        cat "$CACHEFILE"
    else
        script --flush --quiet --return /dev/null --command "$CMD" | tee "$CACHEFILE"
    fi
user_3074130
Related