Cannot remove git refs/original

I have a bare repo I just cleaned out using the following

$ git filter-branch --index-filter \
               'git rm --cached --ignore-unmatch junk/bigfile' -- --all

For a couple of files plus a folder. Attempting to remove second and third files gave me error

Cannot create new backup. A previous backup already exists in refs/original/
Force overwriting the backup with -f

So I attempted

$ git filter-branch -f --index-filter \
               'git rm --cached --ignore-unmatch junk/bigfile' -- --all

for a second file and

$ git filter-branch --index-filter \
               'git rm -rf --cached --ignore-unmatch Folder-I-Want-To-Remove' -- --all

for a folder I wanted to clean out. Now that I'm running

rm -rf refs/original

to attempt to shrink the repository back down I'm getting error

fatal: this operation must be run in a work tree

What should I do? I had planned to run through

$ rm -rf refs/original
$ git fsck
$ git prune
$ git gc
$ git repack -a

But I'm a bit nervous about doing that now since the first command errored out.

Thanks!