Performance issue with AWS Fargate

Viewed 3812

We have an old Java application running in Jboss As 7.1.1. Currently, this application is running on AWS EC2 instance type t3.medium, which is 2 CPU cores and 4GB memory.

We are trying to modernize our applications with CI/CD and AWS Fargate. We succeed to achieve modernization but not with performance.

The task definition of the Fargate has the same CPU and memory as the EC2 instance(2048 CPU and 4096 Memory). The Fargate is in the same subnet and linked to the same security group as the EC2 instance. We couldn't understand the reason for the slow performance.

The application normally searches for music works detail (creators, publishers, artists, and so on). If we take a music work with a large list of interested parties, in the EC2 instance it takes less than 20 seconds but in the Fargate it takes around 30 minutes.

I created the docker image from the same Jboss running on the EC2 instance and the same Java version.

One more piece of information, the database is in a different(non-AWS) data center.

Could you please guide me on how to achieve the same performance as the older setup by using AWS Fargate? What Am I missing in the Fargate?

Task definition

{
    "containerDefinitions": [
      {
        "name": "CONTAINER_NAME",
        "image": "<IMAGE1_NAME>",
        "memory": 4096,
        "essential": true,
        "logConfiguration": {
            "logDriver": "awslogs",
            "secretOptions": null,
            "options": {
              "awslogs-group": "/ecs/TaskDefinitionName",
              "awslogs-region": "eu-central-1",
              "awslogs-stream-prefix": "ecs"
            }
        },

        "portMappings": [
          {
             "hostPort": 8080, 
             "containerPort": 8080,
             "protocol": "tcp"
          }
        ]
      }
    ],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "networkMode": "awsvpc",
    "cpu": "2048",
    "memory": "4096",
    "family": "TaskDefinitionName",
    "taskRoleArn": "arn:aws:iam::accountId:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::accountId:role/ecsTaskExecutionRole"
}

3 Answers

The answer I get from AWS support is as follows.

"ECS Fargate tasks may be placed on any underlying host that is available, and meets the CPU and Memory specified in the task definition and as such may have varying performance. EC2 instances are better suited in situations where control of the application performance is required."

My recommendation for developers, if network throughput and IO performance are important for your application better to use EC2.

While you could expect a small variance in performance between EC2 and Fargate the delta you see is pretty "ridiculous" and there is definitely something going on here. Is the setup 100% identical (with the only difference being where the task runs)? Also, have you tried, for purpose of troubleshooting to start a Fargate task on a larger set of resources (e.g. 4 vCPU / 30GB) to see if that makes any difference? Again just for the sake of figuring out if "capacity" is the issue or the issue lies somewhere else.

The only way to monitor the performance lag is by using Container insights. Only then you can monitor the metrics like CPU and memory utilization, network bandwidth, performance logs. Perhaps then increasing CPU limit and memory may help, in case needed. However Fargate has been reported multiple times to be slower as compared to AWS EC2 being validated by performance tools. So far no concrete solutions has been provided by AWS.

References -

  1. https://discuss.newrelic.com/t/php-agent-on-aws-fargate-performance-issue/124367
  2. https://filia-aleks.medium.com/ec2-versus-fargate-performance-comparison-34b1002fbbaa
  3. https://community.cloudonaut.io/t/spring-boot-on-fargate-slow/318
  4. https://www.reddit.com/r/aws/comments/921szi/migrating_java_applications_from_ec2_to_fargate/
  5. https://www.reddit.com/r/aws/comments/hqjyqd/ecs_performance_10x_slower_than_ec2_instances/
Related