how to use Amazon S3 connector in wso2 ei 6.6 to get all the objects in particular subfolder?

aditi21 0
Viewed 55

I have multiple csv files in object 'bbb' which is in bucket 'aaa'. I am able to download single file as shown below,but now I want download all the files in object 'bbb'

<amazons3.getObject configKey="AMAZON_S3_CONNECTION_1">
<bucketName>aaa</bucketName>
<objectKey>bbb/xxx - mm.dd.yyyy.csv</objectKey>
</amazons3.getObject>

Can someone help me with it. I am using Amazon S3 connector -Version 2.0.3.

I am able to work with getObject for one file, but wanted to see if there is option to bulk download.

Below is the code used <amazons3.init> AMAZON_S3_CONNECTION_1 amazons3 ** </amazons3.init> 

<?xml version="1.0" encoding="UTF-8"?>
<proxy name="AmazonConnectivity" startOnLoad="true" transports="http https vfs" xmlns="http://ws.apache.org/ns/synapse">
    <target>
        <inSequence>
            <log level="full">
                <property name="log" value="====S3 to Local===="/>
            </log>
            <amazons3.getObject configKey="AMAZON_S3_CONNECTION_1">
                <bucketName>aaa</bucketName>
                <objectKey>bbb/xyz.csv</objectKey>
            </amazons3.getObject>
            <log level="full">
                <property name="log" value="====After Amazon S3 Init===="/>
            </log>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </target>
    <parameter name="transport.PollInterval">5</parameter>
    <parameter name="transport.vfs.FileURI">vfs:file:///C:/amazon-s3</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
    <parameter name="transport.vfs.MoveAfterFailure">vfs:file:///C:/failure</parameter>
    <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*.rdy</parameter>
    <parameter name="transport.vfs.MoveAfterProcess">vfs:file:///C:/out</parameter>
</proxy>
wso2-enterprise-integrator wso2-esb wso2
1 Answers

I would suggest you to refer to WSO2 S3 Connector Doc here for reference.

However, an example use case of the getObjectsInBucket that can be used for this scenario is below.

You can set the value in <prefix></prefix> for the targeted sub-folder you want to list.

Sample Request to List:

<getObjectsInBucket>
    <accessKeyId>ACCESS_KEY</accessKeyId>
    <secretAccessKey>ACCESS_SECRET</secretAccessKey>
    <region>REGION</region>
    <delimiter>DELIMITER</delimiter>
    <encodingType>ENCODING_TYPE</encodingType>
    <methodType>GET</methodType>
    <bucketName>aaa</bucketName>
    <bucketUrl>BUCKET_URL</bucketUrl>
    <host>HOST_URL</host>
    <prefix>bbb</prefix>
</getObjectsInBucket>

Edit:

Sample Request to Download:

<getObject>
    <accessKeyId>AKIXXXXXHXQXXG5XX</accessKeyId>
    <secretAccessKey>qHXXBXXXXASYQc4oMCEOj+343HD82s</secretAccessKey>
    <methodType>GET</methodType>
    <contentType>application/xml</contentType>
    <region>us-east-2</region>
    <host>s3.us-east-2.amazonaws.com</host>
    <bucketUrl>http://s3.us-east-2.amazonaws.com/signv4test</bucketUrl>
    <bucketName>signv4test</bucketName>
    <isXAmzDate>true</isXAmzDate>
    <xAmzSecurityToken/>
    <contentMD5/>
    <objectName>Tree2.png</objectName>
    <rangeBytes/>
    <responseContentType/>
    <responseContentLanguage/>
    <responseExpires/>
    <responseCacheControl/>
    <responseContentDisposition/>
    <range/>
    <ifModifiedSince/>
    <ifUnmodifiedSince/>
    <ifMatch/>
    <ifNoneMatch/>
</getObject>

However, downloading multiple object is not supported as for the AWS S3 Docs. There are suggested workarounds as similar to here

userTxZC0
Related