Automated Backups from EC2 to NAS using Rsync

user_6671398 0
Viewed 151

I am trying to automate backups using a raspberry pi with a python script that will rsync everything on the EC2 instances CAMSTEST1 and CAMSProd in the respective backup directory to the on premise NAS.

Hear is the script

#!/usr/bin/python3
import subprocess
from os import path

# private key for AWS Servers
AWS_PRIVATE_KEY = "ARCS-Key-Pair-01.pem"
AWS_USER ="ubuntu"

# NAS backup directory
NAS_BACKUP_DIR = "192.168.1.128:/test"
NAS_MNT = "/mnt/nasper01/"

# CAMSTest1 Config
CAMSTEST1_USER = "ubuntu"
CAMSTEST1_IP = "52.62.119.203"
CAMSTEST1_DIR = "/mnt/backups/*"
CAMSTEST1_MNT = NAS_MNT + "camstest"

#CAMSProd Config
CAMSPROD_USER = "ubuntu"
CAMSPROD_IP = "54.206.28.116"
CAMSPROD_DIR = "/mnt/backups/*"
CAMSPROD_MNT = NAS_MNT + "camsprod"

# mount NAS
print("Mounting NAS")
subprocess.call(["mount","-t", "nfs", NAS_BACKUP_DIR, NAS_MNT])
print("NAS Mounted Successfully")
    
# backup CAMSTEST1
print("Backing Up CAMSTest1")
hostnamefs = "{user}@{ip}:{dir}".format(user=CAMSTEST1_USER,ip=CAMSTEST1_IP,dir=CAMSTEST1_DIR)
sshaccess = 'ssh -i {private_key}'.format(private_key=AWS_PRIVATE_KEY)
subprocess.call(["rsync","-P","-v","--rsync-path","sudo rsync","--remove-source-files","--recursive","-z","-e",sshaccess,"--exclude","/backup-script",hostnamefs, CAMSTEST1_MNT ])
print("Backed Up CAMSTest1 Successfully")

#backup CAMSPROD
print("Backing Up CAMSProd")
hostnamefs = "{user}@{ip}:{dir}".format(user=CAMSPROD_USER,ip=CAMSPROD_IP,dir=CAMSPROD_DIR)
sshaccess = 'ssh -i {private_key}'.format(private_key=AWS_PRIVATE_KEY)
subprocess.call(["rsync","-P","-v","--rsync-path", "sudo rsync","--remove-source-files","--recursive","-z","-e",sshaccess,"--exclude","/backup-script","--exclude","/influxdb-backup", "--exclude", "/db-backup-only",hostnamefs, CAMSPROD_MNT ])
print("Backed Up CAMSProd Successfully")

Hear is the cronjob

0 0 * * 0 sudo python3 /home/pi/backup/backupscript.py >> /home/pi/backup/backuplog

The script works perfectly when run manually from the terminal. However it does not work with a cronjob. It runs without errors but nothing is copied from teh ec2 instacnce to the NAS. Could anyone explain why its not working with a cronjob but is working in the terminal ?

EDIT

Here is the output of the the backups script log with no errors

Last RunTime:
2021-10-31 00:00:02.191447
Mounting NAS
NAS Mounted Successfully
Backing Up ARCSWeb02
Backed up ARCWeb02 Successfully
Backing Up CAMSTest1
Backed Up CAMSTest1 Successfully
Backing Up CAMSProd
Backed Up CAMSProd Successfully
Fetching origin
Last RunTime:
2021-11-07 00:00:02.264703
Mounting NAS
NAS Mounted Successfully
Backing Up ARCSWeb02
Backed up ARCWeb02 Successfully
Backing Up CAMSTest1
Backed Up CAMSTest1 Successfully
Backing Up CAMSProd
Backed Up CAMSProd Successfully
raspberry-pi backup rsync cron python
3 Answers

Rather than a problem with the script itself, which seems to work fine, I would suggest to run the cronjob directly as root by editing the cron file using crontab -e, or by adding a file with root (or other user which can execute the script directly) specified within the job definition into /etc/cron.d/, without specifying the sudo keyword

Here are some references on that matter:

user_155224740

When you run rsync in a subprocess, it picks up a different $PATH from the non-interactive shell; therefore, rsync might not be in its path. Try using the full path to rsync in your subprocess calls.

user_20575090

I have few suggestions,

  1. As suggested before, use cronjob for root user (no sudo)
  2. remove python3 from cronjob, instead make your script executable using `chmod +x ' and run it as
0 0 * * 0 /home/pi/backup/backupscript.py >> /home/pi/backup/backuplog
  1. Just a suggestion, I thing shell script will be more appropriate here, instead of python
user_43871380
Related