AWS DMS Batch apply mode to S3 target

Viewed 1055

I'm migrating on prem MySQL database to AWS S3 using AWS DMS.

I've tried to create cdc replication task which will apply changes from source (mysql) every x amount of time or when buffer gets above certain size. The reason behind that is I want to have big files in S3 for optimized querying. Despite multiple attempts DMS does not batch changes and applies them as they come.

Below is one of the task settings I have tried (I have tried multiple variations):

{
    "TargetMetadata": {
        "BatchApplyEnabled": true
    },
    "ChangeProcessingTuning": {
        "BatchApplyPreserveTransaction": false,
        "BatchApplyTimeoutMin": 3600,
        "BatchApplyTimeoutMax": 3600,
        "MemoryKeepTime": 600
    }
}

I have suspicion that what I'm trying to do is not possible if S3 is the target. I'm basing this on what I've found in CloudFront logs for the task:

2020-03-22T07:49:23 [STREAM_COMPONEN ]I:  Loading stream component 'sorter' with id '101'  (streamcomponent.c:1003)
2020-03-22T07:49:23 [STREAM_COMPONEN ]I:  Loading stream component 'S3' with id '55'  (streamcomponent.c:1003)
2020-03-22T07:49:23 [STREAM_COMPONEN ]I:  Loading stream component 'Mysql' with id '30'  (streamcomponent.c:1003)
2020-03-22T07:49:23 [STREAM_COMPONEN ]I:  Force switch to transactional apply mode for File target  (endpointshell.c:1397)
2020-03-22T07:49:23 [STREAM_COMPONEN ]I:  Loading stream component 'Mysql' with id '30'  (streamcomponent.c:1003)

The Force switch to transactional apply mode for File target makes me think that batch apply mode is not available for S3 target.

1 Answers
Related