I am trying to schedule snapshots for my GCP instances and I just wanted to know is there any possibility that my snapshot data or actual disk data can get corrupted/ OR will be impacted by any way. I mean suppose I am creating snapshot and at the same time some writes are going on how can I ensure that this will not cause any error on my actual disk or in my snapshots?
4 Answers
The answer depends on your operating system.
Microsoft Windows Server support snapshots. Windows will stop file system activity and prepare for the snapshot. This is a safe operation while the OS is running.
Linux does not do anything special when a snapshot is taken (without adding third-party software). This means that snapshots on Linux are NOT safe while the OS is running. Shutdown the OS, then snapshot or image the disk and restart the instance.
Unless an operating system does special things for a snapshot, open files, caching, database handles, etc. are in memory. This data is not part of the snapshot. Therefore, this will cause, in some cases, inconsistent snapshots. This means data corruption.
Google does NOT recommend snapshotting live systems. Neither does AWS or Azure.
Since the snapshotting of a persistent disk is not instantaneous, having writes in parallel could potentially corrupt the operation.
A safe snapshotting process is typically made following these steps:
- Flush your disk's buffers before a snapshot
- Freeze and unmount your filesystem
- Take the snapshot
- Remount the persistent disk
These best practices are directly recommended by Google Cloud and described as follows: https://cloud.google.com/compute/docs/disks/snapshot-best-practices
Yeap, it is recommended to stop your instance before you take the snapshot, but in my experience, if the instance is still running; the I/O from it doesn't seem affected. But the actual information in the snapshot may be corrupted.
Remember to review the quotas of your project on schedule snapshots and the cost will be related around the network traffic.
I let you a link with more info about schedule snapshots: https://cloud.google.com/compute/docs/disks/scheduled-snapshots
No.
There is no impact on Persistent Disk performance while snapshot is taken.
Snapshot will persist all completed IO operations up to a point in time when snapshot was taken. However, depending on application you are using, it may be in inconsistent state. E.g. if you run MySQL with innodb_flush_log_at_trx_commit = 2 you may lose some data as some transactions may not be flushed to PD before they are acknowledged to application.
If you run MySQL with innodb_flush_log_at_trx_commit = 0 (default) then theoretically it should be able to do a crash recovery and restore all committed translations. In practice bugs in applications and filesystems are possible, so I'd advice to test your recovery procedures periodically.
Related