Trigger Fargate to process file uploaded to S3

Question

I am trying to get a CDK application to setup a Fargate task that will automatically process files uploaded to an S3 bucket.

I have a rule target:

rule_target = targets.EcsTask(
    cluster=cluster,
    task_definition=task_definition,
    task_count=1
)

And I have a rule:

s3_upload_rule = events.Rule(
    self,
    's3_upload_rule',
    event_pattern=trigger_event_pattern,
    rule_name='s3_upload_rule',
    targets=[rule_target]
)

There is a lot of other stuff of course, but as far as I can see, the solution must be here. If I were to list everything, this would be a long post. The rule works and the Fargate task is executed, but it doesn't get any input.

My initial assumption would be to use container_overrides just like with a regular run_task, but since it only accepts strings, it would be difficult to tell it that I want the event or parts of the event.

Other events targets take RuleTargetInput parameters which solves this, however, EcsTask does not.

The documentation of Rule says:

targets (Optional[List[IRuleTarget]]) – Targets to invoke when this rule matches an event. Input will be the full matched event. If you wish to specify custom target input, use addTarget(target[, inputOptions]). Default: - No targets.

If I try to do that, I get TypeError: add_target() takes from 1 to 2 positional arguments but 3 were given so the documentation doesn't seem to be correct.

I have also tried creating a RuleTargetConfig with input specification, but I haven't found a way of connecting it to either the rule or the rule target (EcsTask). EcsTask doesn't have any ARN.

Connecting the rule_target to the rule with bind was tempting since it returns a RuleTargetConfig, but the input parameter is read only so that is also a dead end.

I know this is simple to do with lambda and that the lambda handler gets the full event by default, but I need to do heavy processing that will sometimes cause the lambda function to time out. There is of course the possibility of triggering a lambda that will run the Fargate task. This will solve my problem, but then I will keep trying to solve this without lambda anyway since it seems that it must be doable without lambda. Why else would there be a possibility of triggering EcsTasks from EventBridge.

user_4369359 · Answer

Triggering Fargate through EventBridge should work.

This CDK code should do the trick.

events.Rule(
    self,
    bucket_name,
    event_bus=event_bus,
    event_pattern=events.EventPattern(detail_type=['Object Created']),
    targets=[targets.EcsTask(cluster, task_definition, task_definition.task_role)]
)

In order for this to work, you currently have to go to the Properties tab of the S3 bucket in the console and set Amazon EventBridge to On. I have not found a way to do this with CDK.

Trigger Fargate to process file uploaded to S3

1 Answers