How to exclude a specific file in scp linux shell command?

user_5726073 0
Viewed 2528

I am trying to execute the scp command in such a way that it can copy .csv files from source to sink, except a few specific CSV file.

For example in the source folder I am having four files:

file1.csv, file2.csv, file3.csv, file4.csv

Out of those four files, I want to copy all files, except file4.csv, to the sink location.

When I was using the below scp command:

scp /tmp/source/*.csv /tmp/sink/

It would copy all the four CSV files to the sink location.

How can I achieve the same by using the scp command or through writing a shell script?

scp shell linux bash
4 Answers

You can use rsync with the --exclude switch, e.g.

rsync /tmp/source/*.csv /tmp/sink/ --exclude file4.csv
user_2530560

Bash has an extended globbing feature which allows for this. On many installations, you have to separately enable this feature with

shopt -e extglob

With that in place, you can

scp tmp/source/(!fnord*).csv /tmp/sink/

to copy all *.csv files except fnord.csv.

This is a shell feature; the shell will expand the glob to a list of matching files - scp will have no idea how that argument list was generated.

user_8741880

As mentioned in your comment, rsync is not an option for you. The solution presented by tripleee works only if the source is on the client side. Here I present a solution using ssh and tar. tar does have the --exclude flag, which allows us to exclude patterns:

from server to client:

$ ssh user@server 'tar -cf - --exclude "file4.csv" /path/to/dir/*csv' \
  | tar -xf - --transform='s#.*/##' -C /path/to/destination

This essentially creates a tar-ball which is send over /dev/stdout which we pipe into a tar extract. To mimick scp we need to remove the full path using --transform (See U&L). Optionally you can add the destination directory.

from client to server:

We do essentially the same, but reverse the roles:

$ tar -cf - --exclude "file4.csv" /path/to/dir/*csv \
  | ssh user@server 'tar -xf - --transform="s#.*/##" -C /path/to/destination'
user_83440600

You could use a bash array to collect your larger set, then remove the items you don't want. For example:

files=( /tmp/src/*.csv )
for i in "${!files[@]}"; do
  [[ ${files[$i]} = *file4.csv ]] && unset files[$i]
done
scp "${files[@]}" host:/tmp/sink/

Note that our for loop steps through array indices rather than values, so that we'll have the right input for the unset command if we need it.

user_10721120
Related