Concatenate all files of type but specify some files to come first

user_179015 0
Viewed 74

I want to recursively concatenate all the files of a type in a directory structure whilst specifying some files that should come first and some files that should come last.

For example, give I have the following file structure:

header.txt
interesting.txt
sadfsa.txt
oih.txt
nkli.txt
footer.txt
dir/lnln.txt

I would like to specify the concatenation as something like: cat header.txt interesting.txt *.txt footer.txt. But this, would include header.txt - and the other specified files - twice.

What would be the cleanest approach to do this be?

zsh bash
4 Answers

What about storing all these file names in the right order in a bash indexed array?

$ unset a
$ declare -a a=(header.txt interesting.txt sadfsa.txt oih.txt \
  nkli.txt dir/lnln.txt footer.txt)
$ echo "${a[@]}"
header.txt interesting.txt sadfsa.txt oih.txt nkli.txt dir/lnln.txt footer.txt

And then:

$ cat "${a[@]}" > cat.txt

If the intermediate files are all *.txt files under the current directory, except the 3 special ones, you could use find to list them:

$ unset a
$ declare -a a=(header.txt interesting.txt)
$ a+=($(find . -type f ! -name header.txt ! -name interesting.txt ! -name footer.txt -name '*.txt'))
$ a+=(footer.txt)
$ echo "${a[@]}"
header.txt interesting.txt sadfsa.txt oih.txt nkli.txt dir/lnln.txt footer.txt

Finally, if your bash is recent enough (under macOS first install a decently recent one with macports or Homebrew), you can use the extglob option. We assume here that the 3 special files are in the current directory, that there are other txt files in the current directory and in subdirectories:

$ shopt -s extglob
$ echo header.txt interesting.txt !(footer|header|interesting).txt **/*.txt footer.txt
header.txt interesting.txt nkli.txt oih.txt sadfsa.txt dir/lnln.txt footer.txt
user_17737980

A simple solution is to use awk:

$ files=$(printf "%s\n" header.txt interesting.txt **/*.txt | awk "!visited[$0]++")
$ cat $files > output.txt

Explanation:

  • printf "%s\n" header.txt interesting.txt **/*.txt prints all txt files plus the "special" ones each on a separate line. Also works recursively in Bash 4 or newer if you add shopt -s globstar.
  • awk "!visited[$0]++" removes all duplicates. Normally, one could simply use uniq, but sadly that works only for duplicate lines that directly follow each other.
user_149099800

Seeing as you know which files you would like to merge first,

  1. Merge the two files header.txtand interesting.txt with the file you would like to merge with, and have the filenames recorded in an array.

  2. Create a foreach loop, to loop through the files in the directory to be merged, and use an if statement inside the loop, to check if the filename is header.txt or interesting.txt and to continue to the next filename if it is.

user_144450000

You could somehow exclude files from the pattern. One option is to use the caret (^)

cat filewanted1st.txt filewanted2nd.txt *[^td].txt

This way you exclude files which end in a 't' or 'd' before .txt

Unfortunately, this would exclude some other files like filewanted3rd.txt as well (as it ends in a 'd'). So... I don't know better. Maybe someone will know how to use better excluding patterns inside a glob like this.

To find files in subdirectories too, use shopt -s globstar and**/* instead of *:

shopt -s globstar
cat filewanted1st.txt filewanted2nd.txt **/*[^td].txt
user_167253410
Related