Understanding rsync (complex) filters

user_12912013 0
Viewed 54

I'm working on a deployment pipeline via rsync where I have to include/ exclude some special folders and files while there are also a lot of "unknown" files and folders on the destination which I'm not allowed to overwrite or delete.

Let's assume the following simplified structure while +/- means that it should or should not get included:

+ 1/
+ 1/11
+ 1/12
- 1/13
- 1/13/131
+ 1/13/132
+ 1/13/132/1321
+ 2/
+ 2/21
+ 3
- 4
- 5
- everything else

From my understanding, I designed the following rsync:

rsync -rltDvvO --delete --filter='merge .deploy' --dry-run ./ ../dest/

While the file .deploy has the following content:

# allowed folders
+ 1/***
+ 2/***
+ 3

# disallowed folders
- 1/13/***

# override disallowed subfolders to be allowed
+ 1/13/132/***

# disallow everything else
- *

The output is the following:

sending incremental file list
[sender] hiding file .deploy because of pattern *
[sender] showing file 3 because of pattern 3
[sender] showing directory 2 because of pattern 2/***
[sender] hiding file 5 because of pattern *
[sender] hiding file 4 because of pattern *
[sender] showing directory 1 because of pattern 1/***
[sender] showing directory 1/11 because of pattern 1/***
[sender] showing directory 1/13 because of pattern 1/***
[sender] showing directory 1/12 because of pattern 1/***
[sender] showing directory 1/13/131 because of pattern 1/***
[sender] showing directory 1/13/132 because of pattern 1/***
[sender] showing directory 1/13/132/132 because of pattern 1/***
[sender] showing directory 1/13/132/132/1321 because of pattern 1/***
[sender] showing directory 2/21 because of pattern 2/***
delta-transmission disabled for local transfer or --whole-file
3
1/
1/11/
1/12/
1/13/
1/13/131/
1/13/132/
1/13/132/132/
1/13/132/132/1321/
2/
2/21/

For some reason the priority of the rules seems to not be what I intended... is it even possible to filter folders like I need to?

I would like to have only one "rules file". That's why I tried to use filter instead of include-from and exclude-from.

filter rsync
2 Answers

Seems like this one is working but this is really just trial & error, and I don't really understand why it has to be in that order and so on:

# override disallowed subfolders to be allowed
+ 1/13/132/***

# disallowed folders
- 1/13/**

# allowed folders
+ 1/***
+ 2/***
+ 3

# disallow everything else
- *
user_129120130

The lines in the filter file are processed in order. The first line that matches decides: Is it an include, the file will be included, if it is an exclude, it will be excluded.

Let's take a look at your first example filter (I added some comments to the relevant lines):

# allowed folders
+ 1/***           # this will match 1/13, 1/13/131 etc.
+ 2/***
+ 3

# disallowed folders
- 1/13/***        # too late - 1/13/... was already matched above

Now compare it with your second version (from your own answer):

# override disallowed subfolders to be allowed
+ 1/13/132/***                 # does not match 1/13/131

# disallowed folders
- 1/13/**                      # matches 1/13/131, thus it will be excluded

# allowed folders
+ 1/***                        # not relevant any more for 1/13/131
user_57474150
Related