How to connect different Regional S3 Bucket from a Spring Boot Application?

Question

I have a Spring Boot application that has a POST end-point that accepts 2 types of files. Based on the file category, I need to write them to S3 buckets which are in different regions. Example: Category 1 file should be written to Frankfurt (eu-central-1) and Category 2 file should be written to Ohio (us-east-2) S3 buckets. Spring boot accepts a static region (cloud.aws.region.static=eu-central-1) through property configuration and the connection is established when starting the Spring boot so the AmazoneS3 Client Bean is already created with a connection to Frankfurt itself.

I need to containerize this entire setup and deploy it in a K8 Pod.

What is the recommendation for establishing connections and writing objects to different regional buckets? How do I need to implement this? Looking for a dynamic region finding solution rather statically created Bean per region.

Below is a working piece of code that connects to Frankfurt bucket and PUT the object.

@Service
public class S3Service {

    @Autowired
    private AmazonS3 amazonS3Client;
    
    public void putObject(MultipartFile multipartFile) {
        
        ObjectMetadata objectMetaData = new ObjectMetadata();
        objectMetaData.setContentType(multipartFile.getContentType());
        objectMetaData.setContentLength(multipartFile.getSize());

        try {
            PutObjectRequest putObjectRequest = new PutObjectRequest("example-bucket", multipartFile.getOriginalFilename(), multipartFile.getInputStream(), objectMetaData);
            this.amazonS3Client.putObject(putObjectRequest);
        } catch (IOException e) {
            
            /* Handle Exception */
        }
    }
}

Updated Code (20/08/2021)

@Component
public class AmazoneS3ConnectionFactory {

    private static final Logger LOGGER = LoggerFactory.getLogger(AmazoneS3ConnectionFactory.class);
    
    @Value("${example.aws.s3.regions}")
    private String[] regions;
    
    @Autowired
    private DefaultListableBeanFactory beanFactory;
    
    @Autowired
    private AWSCredentialsProvider credentialProvider;
    
    @PostConstruct
    public void init() {
        
        for(String region: this.regions) {
            
            String amazonS3BeanName = region + "_" + "amazonS3";
            
            if (!this.beanFactory.containsBean(amazonS3BeanName)) {
                
                AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withPathStyleAccessEnabled(true)
                        .withCredentials(this.credentialProvider).withRegion(region).withChunkedEncodingDisabled(true);
                AmazonS3 awsS3 = builder.build();
                this.beanFactory.registerSingleton(amazonS3BeanName, awsS3);
                
                LOGGER.info("Bean " + amazonS3BeanName + " - Not exist. Created a bean and registered the same");
            }
        }
        
    }
    
    /**
     * Returns {@link AmazonS3} for a region. Uses the default {@link AWSCredentialsProvider}
     */
    public AmazonS3 getConnection(String region) {
        
        String amazonS3BeanName = region + "_" + "amazonS3";
        return (AmazonS3Client)this.beanFactory.getBean(amazonS3BeanName, AmazonS3.class);
    }
}

My Service layer will call the "getConnection()" and get the AmazonS3 Object to operate on it.

user_16572295 · Answer

The only option that I am aware is to create different S3Client with S3ClientBuilder, one for each different region. You would need to register them as Spring Beans with different names so that you can later autowire them.

Update (19/08/2021)

The following should work (sorry for the Kotlin code but it is faster to write):

Class that may contain your configuration for each region.

class AmazonS3Properties(val accessKeyId: String,
                     val secretAccessKey: String,
                     val region: String,
                     val bucket: String)

Configuration for S3 that will create 2 S3Clients and stored the buckets for each region (later needed).

@Configuration
class AmazonS3Configuration(private val s3Properties: Map) {

  lateinit var buckets: Map

  @PostConstruct
  fun init() {
    buckets = s3Properties.mapValues { it.bucket }
  }

  @Bean(name = "regionA")
  fun regionA(): S3Client {
    val regionAProperties = s3Properties["region-a"]
    val awsCredentials = AwsBasicCredentials.create(regionAProperties.accessKeyId, regionAProperties.secretAccessKey)
    return S3Client.builder().region(Region.of(regionAProperties.region)).credentialsProvider { awsCredentials }.build()
  }

  @Bean(name = "regionB")
  fun regionB(): S3Client {
    val regionBProperties = s3Properties["region-b"]
    val awsCredentials = AwsBasicCredentials.create(regionBProperties.accessKeyId, regionBProperties.secretAccessKey)
    return S3Client.builder().region(Region.of(regionBProperties.region)).credentialsProvider { awsCredentials }.build()
  }
}

Service that will target one of the regions (Region A)

@Service
class RegionAS3Service(private val amazonS3Configuration: AmazonS3Configuration,
                       @field:Qualifier("regionA") private val amazonS3Client: S3Client) {

  fun save(region: String, byteArrayOutputStream: ByteArrayOutputStream) {
      val inputStream = ByteArrayInputStream(byteArrayOutputStream.toByteArray())
      val contentLength = byteArrayOutputStream.size().toLong()

      amazonS3Client.putObject(PutObjectRequest.builder().bucket(amazonS3Configuration.buckets[region]).key("whatever-key").build(), RequestBody.fromInputStream(inputStream, contentLength))
  }
}

How to connect different Regional S3 Bucket from a Spring Boot Application?

1 Answers