This issue is related with Terraform 0.12 version
Hi! I'm currently using Terraform to deploy an AWS EMR cluster in my team and every time we add new resources, Terraform recreates the cluster. The declaration of our cluster is the following:
resource "aws_emr_cluster" "emr_spark_cluster" {
name = "whatever_name_to_emr_spark_cluster"
release_label = "emr-5.29.0"
applications = ["Spark"]
log_uri = "whatever_log_uri"
ec2_attributes {
subnet_id = var.vpc_public_subnets[0]
emr_managed_master_security_group = aws_security_group.emr_sg.id
emr_managed_slave_security_group = aws_security_group.emr_sg.id
instance_profile = aws_iam_instance_profile.instance_profile.arn
key_name = "whatever_name_to_ec2_attributes"
}
master_instance_group {
instance_type = var.emr_master_instance_type
}
core_instance_group {
instance_type = var.emr_core_instance_type
instance_count = var.emr_core_instance_count
ebs_config {
size = var.emr_core_ebs_size
type = whatever_type
volumes_per_instance = whatever_volumes_per_instance
}
}
tags = var.tags
service_role = aws_iam_role.emr_service.arn
}
Basically we're using the master_instance_group and the core_instance_group as the Terraform 0.12 docs suggest (1)
With the cluster already deployed and trying to add new resources, the terraform apply command says that the cluster needs to be recreated (removed all non interesting stuff):
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.emr.aws_emr_cluster.emr_spark_cluster must be replaced
-/+ resource "aws_emr_cluster" "emr_spark_cluster" {
applications = [whatever_applications]]
- ebs_root_volume_size = 0 -> null
- instance_group {
- id = whatever_id -> null
- instance_count = 2 -> null
- instance_role = "CORE" -> null
- instance_type = whatever_the_instance_type -> null
- ebs_config {
- iops = 0 -> null
- size = whatever_size -> null
- type = whatever_type -> null
- volumes_per_instance = whatever_volumes_per_instance -> null
}
}
- instance_group {
- id = whatever_id -> null
- instance_count = whatever_instance_count -> null
- instance_role = "MASTER" -> null
- instance_type = whatever_the_instance_type -> null
- ebs_config {
- iops = 0 -> null
- size = whatever_size -> null
- type = whatever_type -> null
- volumes_per_instance = whatever_volumes_per_instance -> null
}
}
+ instance_group {
+ autoscaling_policy = (known after apply)
+ bid_price = (known after apply)
+ id = (known after apply)
+ instance_count = (known after apply)
+ instance_role = (known after apply)
+ instance_type = (known after apply)
+ name = (known after apply)
+ ebs_config {
+ iops = (known after apply)
+ size = (known after apply)
+ type = (known after apply)
+ volumes_per_instance = (known after apply)
}
}
~ master_instance_group {
~ id = whatever_id -> (known after apply)
instance_count = 1
instance_type = whatever_the_instance_type
- ebs_config {
- iops = 0 -> null
- size = whatever_size -> null
- type = whatever_type -> null
- volumes_per_instance = whatever_volumes_per_instance -> null
}
+ ebs_config {
+ iops = (known after apply)
+ size = (known after apply)
+ type = (known after apply)
+ volumes_per_instance = (known after apply)
}
}
}
Basically the part that I'm really interested is in the instance_group. I suppose that's the reason why the cluster needs to be recreated because the current master_instance_group and the core_instance_group properties don't "reserve" specific instance groups but only the type and size of them, right?
For that reason as a workaround I tried using the aws_emr_instance_group resource trying to substitute the master_instance_group and the core_instance_group properties and link them directly to the cluster by the cluster_id property. As an example, the aws_emr_instance_group of the master_instances:
resource "aws_emr_instance_group" "master_instances" {
name = "emr_cluster_master_instances"
cluster_id = aws_emr_cluster.emr_spark_cluster.id
instance_type = var.emr_master_instance_type
instance_count = var.emr_master_instance_count
ebs_config {
iops = 0
size = var.emr_core_ebs_size
type = whatever_type
volumes_per_instance = whatever_volumes_per_instance
}
}
But at the moment of execute the terraform plan it fails with the following error:
module.emr.aws_emr_cluster.emr_spark_cluster: Creating...
Error: error running EMR Job Flow: ValidationException: Instance count must be greater than 0
I don't understand the error because I'm linking the instances group with the cluster.
So my questions are:
- How can I set specific instances to my cluster in order to avoid a recreation of it every time I need to add new resources (deploy new resources in another pipeline is discarded). Do I have to use
master_instance_type,core_instance_typeandcore_instance_counteven when those are deprecated params? Maybe I'm using wrongly theaws_emr_instance_groupresource? - Can it be possible that maybe that's the expected behaviour with the properties
master_instance_groupand thecore_instance_group? I mean, to avoid create specific instances to be active even when the cluster is destroyed?
Thanks in advance
(1) I say "suggest" because other properties like core_instance_type, core_instance_count and master_instance_type are in Terraform 0.12 deprecated.