How do I sync changes on AWS S3 (ultimately on AWS EFS) back into a Github repository?

user_2113148 0
Viewed 872

I use Github Actions to sync one of my git repo's subfolders (including < 200 very small MD and YAML files) to AWS S3. I then use AWS Lambda to further sync this S3 Bucket with AWS EFS.

How do I synchronise the S3 (or even better - the EFS) bucket with added/changed/deleted files back to my git repository on Github?

  • I haven't been able to find any Github Actions that would offer this functionality
  • As I use EC2 to mount my EFS, I thought that I might git clone and create a symbolic link to the EFS. I could then skip the S3 side-step and invoke git pull/push.
  • I thought I could build a Lambda that would spin up a EC2 instance that would automatically pull those files from S3 and push a commit to my git repository.
  • I could try to figure out how to use Lambda and the /tmp to formulate a commit

I haven't been able to wrap my head around what would the best way to achieve that.

Can you please push me in the right direction?

P.S. I'm not very proficient with git, this whole implementation it's a bit out of my league. :-)

amazon-efs aws-lambda github-actions amazon-s3 git
2 Answers

@vonc, thank you for your hint, it moved me in the right direction.

I ended up doing this, which solves my ultimate goal, which is to manage EFS through GIT. I can fully skip S3 / Lambda.

Step 1: sudo mount -t efs -o tls MYFILESYSTEMID:/ efs

Step 2: cd efs

Step 3: git clone --depth 1 --no-checkout --filter=blob:none git@github.com:user/repo

Step 4: cd repo

Step 5: git sparse-checkout set subdirectory

Step 6: git read-tree -mu HEAD

From here on, I can push/pull changes in this particular subdirectory.

user_21131480

If you can clone your Git repository on the same EC2 instance when you have mounted your AWS EFS, you can do:

cd /path/to/cloned/git/repository
git --work-tree=/path/to/mounted/EFS add .

That will detect added/modified/deleted changes from EFS and import those changes to your cloned Git repository.
From there, you can commit and push.

VonC_63090
Related