Read a file using Java from an S3 bucket and HTTP PUT file to presigned AWS S3 URL of another bucket in a way that simulates an actual file upload

Viewed 1564

New to Java and HTTP requests.

Why this question is not a duplicate: I'm not using AWS SDK to generate any presigned URL. I get it from an external API.

Here is what I'm trying to accomplish:

Step 1: Read the source S3 bucket for a file (for now .xlsx)

Step 2: Parse this file by converting it to an InputStreamReader (I need help here)

Step 3: Do a HTTP PUT of this file by transferring the contents of the InputStreamReader to an OutputStreamWriter, on a pre-signed S3 URL that I already have obtained from an external team. The file must sit in the destination S3 bucket, in the exact way a file is uploaded manually by dragging and dropping. (Also need help here)

Here is what I've tried:

Step 1: Read the S3 bucket for the file

public class LambdaMain implements RequestHandler<S3Event, String>  {

    @Override
    public String handleRequest(final S3Event event, final Context context) {

        System.out.println("Create object was called on the S3 bucket");
        S3EventNotification.S3EventNotificationRecord record = event.getRecords().get(0);

        String srcBucket = record.getS3().getBucket().getName();
        String srcKey = record.getS3().getObject().getUrlDecodedKey();

        AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
                .build();

        S3Object s3Object = s3Client.getObject(new GetObjectRequest(
                srcBucket, srcKey));

        String presignedS3Url = //Assume that I have this by making an external API call
        InputStreamReader inputStreamReader = parseFileFromS3(s3Object); #Step 2
        int responseCode = putContentIntoS3URL(inputStreamReader, presignedS3Url); #Step 3

}

Step 2: Parse the file into an InputStreamReader to copy it to an OutputStreamWriter:

    private InputStreamReader parseFileFromS3(S3Object s3Object) {
        return new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
    }

Step 3: Make a HTTP PUT call by copying the contents from InputStreamReader to OutputStreamWriter:

   private int putContentIntoS3URL(InputStreamReader inputStreamReader, String presignedS3Url) {
        URL url = null;
        try {
            url = new URL(presignedS3Url);
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }
        HttpURLConnection httpCon = null;

        try {
            assert url != null;
            httpCon = (HttpURLConnection) url.openConnection();
        } catch (IOException e) {
            e.printStackTrace();
        }
        httpCon.setDoOutput(true);

        try {
            httpCon.setRequestMethod("PUT");

        } catch (ProtocolException e) {
            e.printStackTrace();
        }

        OutputStreamWriter outputStreamWriter = null;
        try {
            outputStreamWriter = new OutputStreamWriter(
                    httpCon.getOutputStream());
        } catch (IOException e) {
            e.printStackTrace();
        }

        try {
            IOUtils.copy(inputStreamReader, outputStreamWriter); 
        } catch (IOException e) {
            e.printStackTrace();
        }

        try {
            outputStreamWriter.close();
        } catch (IOException e) {
            e.printStackTrace();
        }

        try {
            httpCon.getInputStream();
        } catch (IOException e) {
            e.printStackTrace();
        }

        int responseCode = 0;

        try {
            responseCode = httpCon.getResponseCode();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return responseCode;
    }

The issue with the among approach is that when I read an .xlsx file via an S3 insert trigger and PUT into the URL, when I download the uploaded file - it gets downloaded as some gibberish.

When I try reading in a .png file and PUT into the URL, when I download the uploaded file - it gets downloaded as some text file with some gibberish (I did see the word PNG in it though)

It feels like I'm making mistakes with:

  1. Incorrectly creating an OutputStreamWriter since I don't understand how to send a file via a HTTP request

  2. Assuming that every file type can be handled in a generic way.

  3. Not setting the content-type in the HTTP request

  4. Expecting S3 to magically understand my file type after the PUT operation

I would like to know if my above 4 assumptions are correct or incorrect.

The intention is that, I do the PUT on the file data correctly so it sits in the S3 bucket along with the correct file type/extension. I hope my effort is worthy to garner some help. I've done a lot of searching into HTTP PUT and File/IO, but I'm unable to LINK them together for my use-case, since I perform a File I/O followed by a HTTP PUT.

UPDATE 1:

I've added the setRequestProperty("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), but the file doesn't sit in the S3 bucket with the file extension. It simply sits there as an object.

UPDATE 2:

I think this also has something to do with setContentDisposition() header, although I'm not sure how I go about setting these headers for Excel files.

UPDATE 3:

This may simply have to do with how the Presigned S3 URL itself is vended out to us. As mentioned in the question, I said that we get the Presigned S3 URL from some other team. The question itself has multiple parts that need answering.

  1. Does the default Presigned S3 URL ALLOW clients to set the content-type and content-disposition in the HTTP header?: I've set up another separate question here since it's quite unclear: Can a client set file name and extension programmatically when he PUTs file content to a presigned S3 URL that the service vends out?

  2. If the answer to above question is TRUE, then and only then must we go into how to set the file contents and write it to the OutputStream

1 Answers

You are using InputStreamReader and OutputStreamWriter, which are both bridges between a byte stream and a character stream. However, you are using these with byte data, which means you first convert your bytes to characters, and then back to bytes. Since your data is not character data, this conversion might explain why you get gibberish as a result.

I'd start trying to get rid of the reader and writer, instead directly using the InputStream (which you already got from s3Object.getObjectContent()), and the OutputStream (which you got from httpCon.getOutputStream()). IOUtils.copy should also support this.

Also as a side note, when you construct the InputStreamReader you set StandardCharsets.UTF_8 as the charset to use, but when you construct the OutputStreamWriter you don't set the charset. Should the default charset not be UTF-8, this conversion would probably also result in gibberish.

Related