How to upload folders/files from Git repositories to S3 bucket with Terraform?

Unfortunately, the GitHub Terraform provider does not provide any data sources for getting a list of files in a repository or for retrieving individual files.

Therefore, if the repository is private, the only feasible way to do this is to clone the repository to the machine where Terraform is running, then use the fileset function to get a list of files in the repository, then use that list in a for_each on the aws_s3_bucket_object resource to upload each file. You'll need to do the cloning in an external shell command; you can use a module such as this one to accomplish that.

If the repository is public, and you know ahead of time which files you want to send to S3, then you can use the HTTP provider to download the file from its GitHub URL, parse the response body to extract the file contents, and then specify those contents in the content field of the aws_s3_bucket_object resource.

Since October 2021 the GitHub Terraform provider has the data source github_repository_file. This data source allows you to read files within a GitHub repository. You can then upload your file to S3 using the aws_s3_object resource from the Terraform AWS provider.

provider "aws" {
  region     = "us-east-1"
  access_key = "ABC123"
  secret_key = "dEf456"
}
 
provider "github" {
  token = "xyz890"
  owner = "some-owner"
}
 
data "github_repository_file" "my_file" {
  provider   = github
  repository = "my-repository"
  file       = "my_file.py"
}
 
resource "aws_s3_object" "my_file_on_s3" {
  bucket  = "my-bucket"
  key     = "my_filename_on_s3"
  content = data.github_repository_file.my_file.content
}