I am creating a cloud formation template to provision elasticsearch service domain in AWS.
I would like to set this property under Encryption to true "Require HTTPS for all traffic to the domain" but I am not able to find the way in AWS docs to do so.
Other options for setting encryption properties like "Enable encryption of data at rest" & "Node-to-node encryption" are well documented.
Does anyone know how to set "Require HTTPS for all traffic to the domain" property from CF template ?
The way I did this was to make sure the security group only allows HTTPS (443) access to the cluster.
Not completely sure if this is what you are looking for, but if it's not, give me some more details and I will see if I can help you out.
mySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VpcId
GroupName: !Ref SecurityGroup
GroupDescription: !Ref GroupDescription
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Here is what I have put together for Encryption at Rest as well as Node to node encryption:
Now, I had to add/edit plenty of moving parts, so I will post the whole template here so you can see what I did in the different sections so you can pull out what you need/want. I used Conditionals since you can chose to enable it or not. I also made sure to add a few comments here and there. Feel free to strip those if you want, but they do not impact the template at all.
edit: I know that there are several parts of the template that are not NEEDED, but I like to make mine look pretty and organized (when being used). I know I need to work on organization inside the template itself, but it's all there. :P
edit2: The below template assumes you will be using Existing VPC/Subnets/Security Groups, you can check out my repo (in comment below) for other versions of this template.
I did validate this template and was able to successfully build the Domain.
AWSTemplateFormatVersion: 2010-09-09
Description: >-
**NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
verify you have the following Service Role created in IAM!!
-- AWSServiceRoleForAmazonElasticsearchService --
If you do not have this Role, create it using the following CLi Command --
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Configure Cluster"
Parameters:
- DomainName
- ElasticsearchVersion
- ZoneAwareness
- SnapShotHour
-
Label:
default: "Data Instances"
Parameters:
- InstanceType
- DataInstanceCount
-
Label:
default: "Dedicated Master Instances"
Parameters:
- DedicatedMaster
- MasterInstanceType
- MasterInstanceCount
-
Label:
default: "Storage Config"
Parameters:
- StorageSize
-
Label:
default: "Network Config"
Parameters:
- VpcId
- SubNet1
- SubNet2
- SecurityGroup
-
Label:
default: "Encryption Settings"
Parameters:
- EncryptionAtRest
- KmsKey
- NodetoNode
-
Label:
default: "IAM User Restriction Policy"
Parameters:
- IamUserArn
ParameterLabels:
DomainName:
default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
ElasticsearchVersion:
default: "Select the ElasticSearch version desired"
InstanceType:
default: "Instance Size of Data Instances"
DataInstanceCount:
default: "Number of Data Instances Required"
DedicatedMaster:
default: "Select if a Dedicated Master Instance is required"
MasterInstanceType:
default: "Instance Size of Master Instances"
MasterInstanceCount:
default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
StorageSize:
default: "Storage Size in GB"
VpcId:
default: "Select the VPC to deploy into (must already exist)"
SubNet1:
default: "Select the First Subnet"
SubNet2:
default: "Select the Second Subnet"
SecurityGroup:
default: "Select the Security Group"
IamUserArn:
default: "Enter the ARN for the IAM User to give initial access to the stack"
ZoneAwareness:
default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
SnapShotHour:
default: "Set the hour to run the Automated Snapshot (0-23) (Default: UTC Timezone)"
EncryptionAtRest:
default: "Enable Encryption at Rest"
KmsKey:
default: "If Encryption at Rest is enabled, supply the KMS Key used for encryption"
NodetoNode:
default: "Enable Node to Node Encryption"
Parameters:
DomainName:
Type: String
Default: "elasticsearchstack-cf"
MaxLength: '128'
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
SnapShotHour:
Type: Number
Default: 0
MinValue: 0
MaxValue: 23
ElasticsearchVersion:
Type: String
Default: 7.1
AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
InstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
DataInstanceCount:
Type: Number
Default: 2
AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
MasterInstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
MasterInstanceCount:
Type: Number
Default: 0
AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
VpcId:
Type: AWS::EC2::VPC::Id
ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
SubNet1:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SubNet2:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
ConstraintDescription: "Must be and existing Security Group."
DedicatedMaster:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
StorageSize:
Type: Number
Default: 20
MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
MaxValue: 1000 # Remove this line for free-form number entry
IamUserArn:
Type: String
Default: "arn:aws:iam::<AccountNumber>:user/<username>"
ZoneAwareness:
Description: True or False
Type: String
Default: True
AllowedValues:
- True
- False
EncryptionAtRest:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
KmsKey:
Type: String
NodetoNode:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
Conditions: # Checks to see if Conditional Values are True
DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
EncryptionAtRestYes: !Equals [ !Ref EncryptionAtRest, True]
Resources:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref DomainName
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
DedicatedMasterEnabled: !Ref DedicatedMaster
InstanceCount: !Ref DataInstanceCount
ZoneAwarenessEnabled: !Ref ZoneAwareness
InstanceType: !Ref InstanceType
DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
DedicatedMasterCount:
!If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
EBSOptions:
EBSEnabled: True
Iops: 0
VolumeSize: !Ref StorageSize
VolumeType: "gp2"
SnapshotOptions:
AutomatedSnapshotStartHour: !Ref SnapShotHour
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: !Ref IamUserArn
Action: 'es:*'
Resource: "*"
AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
VPCOptions:
SubnetIds:
- !Ref SubNet1
- !Ref SubNet2
SecurityGroupIds:
- !Ref SecurityGroup
EncryptionAtRestOptions: # If Encryption At Rest is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [EncryptionAtRestYes, !Ref EncryptionAtRest, !Ref "AWS::NoValue"]
KmsKeyId:
!If [EncryptionAtRestYes, !Ref KmsKey, !Ref "AWS::NoValue"]
NodeToNodeEncryptionOptions:
Enabled: !Ref NodetoNode
Outputs:
DomainArn:
Value: !GetAtt ElasticsearchDomain.DomainArn
DomainEndpoint:
Value: !GetAtt ElasticsearchDomain.DomainEndpoint
SecurityGroupId:
Value: !Ref SecurityGroup
SubnetId1:
Value: !Ref SubNet1
SubnetId2:
Value: !Ref SubNet2
As feature was recently released, it is not yet available in CloudFormation or Terraform, But we can achieve same by using custom resource in a CF template. This custom resource is nothing but a lambda function which will be executed as part cloud formation stack creation.
complete automation can be divided into 3 section,
As explained here follow below steps to create lambda fuction.
Create an empty folder that you’ll use to place your Lambda source. Then use pip to install crhelper into the folder, and create the lambda_function.py file to put your resource code into.
mkdir sum_function
cd sum_function
pip install -t . crhelper
# on some systems pip may fail with a distutils error, if you run into this, try
running pip with the –system argument
# pip install —system -t . crhelper
touch lambda_function.py
Now open the lambda_function.py file and place the following code into the lambda_function.py file.
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger(__name__)
helper = CfnResource()
@helper.create
@helper.update
def update_es_domain_config(event, context):
logger.info("Updating elasticsearch domain config with HTTPS")
es = boto3.client('es')
response = es.update_elasticsearch_domain_config(
DomainName=event['ResourceProperties']['ESDomainName'],
DomainEndpointOptions={
'EnforceHTTPS': True,
'TLSSecurityPolicy': 'Policy-Min-TLS-1-0-2019-07'
}
)
logger.info("results:", response)
@helper.delete
def no_op(_, __):
pass
def handler(event, context):
helper(event, context)
After that zip the whole directory
zip -r ../sum.zip ./
Now create the lambda function, I am using CLI for the same. Replace the values according to your environment.
aws lambda create-function \
--function-name "update-es-domain" \
--handler "lambda_function.handler" \
--timeout 900 \
--zip-file fileb://../sum.zip \
--runtime python3.7 \
--role "arn:aws:iam::123412341234:role/lambda-cli-role"
You’ll need to take down the FunctionArn from the output to use in the next step.
Now In Resources section of your CF template, after Elasticsearch resource, add custom resource as shown below. Replace the service token with the functionArn you get in last step as a output. Replace ESDomainName with the elasticsearch domain that you want to modify.
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
PFB Complete Cloudformation template for reference.
AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation templates to create AWS Elasticsearch Service domain.
Parameters:
NodeType:
Description: The node type to be provisioned for the Elasticsearch cluster
Type: String
Default: m4.large.elasticsearch
AllowedValues:
- m4.large.elasticsearch
- m4.xlarge.elasticsearch
- c4.large.elasticsearch
- c4.xlarge.elasticsearch
- r4.large.elasticsearch
- r4.xlarge.elasticsearch
ConstraintDescription: must be a valid Elasticsearch node type.
NodeCount:
Description: The number of nodes in the Elasticsearch cluster.
Type: Number
Default: '1'
AllowedESGetIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch retrieval
AllowedESUpdateIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch update
ESDomainName:
Type: String
Description: Name of Elastic Search Domain
Default: advantagehelp
Resources:
ElasticsearchDomain:
Type: 'AWS::Elasticsearch::Domain'
Metadata:
cfn_nag:
rules_to_suppress:
- id: W28
reason: "Needed to name elasticsearch domain"
Properties:
DomainName: !Ref ESDomainName
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: !Ref NodeCount
ZoneAwarenessEnabled: false
InstanceType: !Ref NodeType
ElasticsearchVersion: '6.4'
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESGetIpList
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
- 'es:ESHttpPost'
- 'es:ESHttpPut'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESUpdateIpList
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: gp2
SnapshotOptions:
AutomatedSnapshotStartHour: 1
AdvancedOptions:
rest.action.multi.allow_explicit_index: "true"
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
DependsOn: ElasticsearchDomain
Outputs:
KibanaURL:
Description: Kibana URL
Value: !Join
- ''
- - !GetAtt
- ElasticsearchDomain
- DomainEndpoint
- /_plugin/kibana/
ElasticsearchEndpoint:
Description: Elasticsearch domain endpoint
Value: !GetAtt
- ElasticsearchDomain
- DomainEndpoint
ElasticsearchDomainARN:
Description: Elasticsearch domain ARN
Value: !GetAtt
- ElasticsearchDomain
- DomainArn