Scaleway GLACIER class object storage with restic

user_965586 0
Viewed 1274

Scaleway recently launched GLACIER class storage "C14 Cold Storage Class"

They have a great plan of 75GB free and I'd like to take advantage of this using the restic backup tool.

To get this working I have successfully followed the S3 instructions for repository creation and uploading, with one caveat. I can not successfully pass the storage-class header as GLACIER.

Using awscliv2, I can successfully pass a header that looks very much like this from my local machine: aws s3 cp object s3://bucket/ --storage-class GLACIER

But with restic, having dug through some github issues, I can see an option to pass a -o flag. The linked issues resolution is not that clear to me so I have tried the following restic commands without successfully seeing the "GLACIER" class of storage label next to the files objects in the Scaleway bucket console:

  • restic -r s3:s3.fr-par.scw.cloud/restic-testing -o GLACIER --verbose backup ~/test.txt
  • restic -r s3:s3.fr-par.scw.cloud/restic-testing -o storage-class=GLACIER --verbose backup ~/test.txt

Can someone suggest another option?

backup
2 Answers

I'm starting to use C14's GLACIER storage class with restic, and until now it seems be working very well.

I suggest to create the repository in the usual way with restic -r s3:s3.fr-par.scw.cloud/test-bucket init, which will create the config file and keys in the STANDARD storage class. For backups, I'm using the command:

$ restic backup -r s3:s3.fr-par.scw.cloud/test-bucket -o s3.storage-class=GLACIER --host host /path

similar to what you did, apart the option is s3.storage-class and not storage-class.

In this way files in the data and snapshots directories are in GLACIER storage class, and you can add backups with no problem. I can also mount the repository while data is in GLACIER class (I suppose all the info are taken from cache) so I can do restic mount /mnt/c14 and I can browse the files, also if I cannot copy them or see their content.

If I need to restore files, I restore all bucket in STANDARD class with s3cmd restore --recursive s3://test-bucket/ (see s3cmd), I test that all files are correctly in standard class with:

$ aws s3 ls s3://test-bucket --recursive | tr -s ' ' | cut -d' ' -f 4 | xargs -n 1 -I {} sh -c "aws s3api head-object --bucket unitedhost --key '{}' | jq -r .StorageClass" | grep --quiet GLACIER

which returns true if at least one file is in GLACIER class, so you have to wait this command to returns false.

Obviously a restore will need more time, but I'm using C14 glacier as a second or third backup, while using another restic repository in Backblaze B2 which is a warm storage.

user_38212380

In addition to vstefanoxx 's answer : Here is my workflow.

I setup the restic repository just like vstefanoxx.

Now, if you want to prune the repository... you cannot as the files are in glacier and restic needs read-write access to the bucket to prune. What is interesting about Scaleway is that file transferts between glacier and standard class are free. So let's move the data back to the standard class :

s3cmd restore --recursive s3://test-bucket

And wait until the end of the process using the command given by vstefanoxx. Once your data is in the standard class it costs you five times more, so we have to be efficient :-)

So we now prune the repository: restic prune -r s3:s3.fr-par.scw.cloud/test-bucket

And once it is finished, move everything (in fact data, index and snapshots but not keys) back to glacier:

s3cmd cp s3://test-bucket/data/ s3://test-bucket/data/ --recursive --storage-class=GLACIER
s3cmd cp s3://test-bucket/index/ s3://test-bucket/index/ --recursive --storage-class=GLACIER
s3cmd cp s3://test-bucket/snapshots/ s3://test-bucket/snapshots/ --recursive --storage-class=GLACIER

So we are now to a point where we have pruned the repository, trying to pay the least amount of money !

user_82517050
Related