I've been tasked with tuning the snapshotting process. I'm dealing with 3 master node instances and 9 data node instances. We are using S3 as the store for the repositories.
There are 28 indices and total of 189 shards in this particular cluster. The only snapshot tuning parameters I can find are chunk_size and max_snapshot_bytes_per_sec.
I've left chunk_size at the default (unlimited) and modified the bytes_per_sec and from the default (40mb) to 100mb and 500mb respectively.
Baseline it takes 3 hours to snapshot the entire cluster (using the default settings), after two experiments with changes in just bytes_per_sec, the snapshot process is still 3 hours.
To me this sounds like the process is either CPU or network bound or am I missing something? Not sure what other parameters I can change.