How to make two EC2 instance (installed Elasticsearch created by AMI) as multi-node using cloudformation template?

Viewed 264

I need to create two Ec2 instances using AMI and make it as multi-node using CloudFormation template. The AMI installed the elasticsearch in it. I need to make one a master node and another one a data node.

My CF template script,

AWSTemplateFormatVersion: '2010-09-09'
#Transform: 'AWS::Serverless-2016-10-31'
Description: AWS CloudFormation Template with EC2InstanceWithSecurityGroup
Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  RemoteAccessLocation:
    Description: The IP address range that can be used to access to the EC2 instances
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.

Resources:
  ES1EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.2xlarge
      SecurityGroups:
        - !Ref 'InstanceSecurityGroup'
      KeyName: !Ref 'KeyName'
      ImageId: ami-xxxxxxxxxxxxxxxx
      #DependsOn: ES2EC2Instance
      UserData:
        Fn::Base64: !Sub |
              #!/bin/bash -ex

              cat > /etc/elasticsearch/elasticsearch.yml<<EOF1
              network.host: "${EC2_PRIVATE_IP}"
              http.port: 9200
              http.max_content_length: 1gb
              node.name: node-1
              node.roles: [ master, data, ingest ]
              transport.port: 9300-9400
              discovery.seed_hosts: ["${ES1EC2Instance.PrivateIp}", "${ES2EC2Instance.PrivateIp}"]
              cluster.initial_master_nodes: ["node-1"]
              gateway.recover_after_nodes: 2
              EOF1

              ## Restart Elasticsearch
              sudo systemctl restart elasticsearch
  ES2EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.2xlarge
      SecurityGroups:
        - !Ref 'InstanceSecurityGroup'
      KeyName: !Ref 'KeyName'
      ImageId: ami-xxxxxxxxxxxxxxxx
      #DependsOn: ES1EC2Instance
      DependsOn: ES1EC2Instance
      UserData:
        Fn::Base64: !Sub |
              #!/bin/bash -ex

              cat > /etc/elasticsearch/elasticsearch.yml<<EOF1
              network.host: "${ES2EC2Instance.PrivateIp}"
              http.port: 9200
              http.max_content_length: 1gb
              node.name: node-2
              node.roles: [ data, ingest ]
              transport.port: 9300-9400
              discovery.seed_hosts: ["${ES1EC2Instance.PrivateIp}", "${ES2EC2Instance.PrivateIp}"]
              cluster.initial_master_nodes: ["node-1"]
              gateway.recover_after_nodes: 2
              EOF1

              ## Restart Elasticsearch
              sudo systemctl restart elasticsearch

  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH (22), HTTP (8080),
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: !Ref 'RemoteAccessLocation'
        - CidrIp: 0.0.0.0/0
          FromPort: '8080'
          IpProtocol: tcp
          ToPort: '8080'
        - IpProtocol: tcp
          FromPort: '9200'
          ToPort: '9200'
          CidrIp: !Ref 'RemoteAccessLocation'

Outputs:
  AZ:
    Description: Availability Zone of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.AvailabilityZone'
  PublicDNS:
    Description: Public DNSName of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.PublicDnsName'
  PublicIP:
    Description: Public IP address of the newly created EC2 instance for ES
    Value: !GetAtt 'ES1EC2Instance.PublicIp'

How can I update the elasticsearch.yaml to make it as a multi-node using CloudFormation Template?

1 Answers

Try to add a macro to your Cloudformation template. This is an example of lambda that can be invoked by a macro. This function use SSM RunCommand to send bash commands to your instance, at this case the instances are filtered by autoscaling groups, but you can filter your instances by tags or any other attribute. Also you need to add a IAM role to the instance with the AmazonEC2RoleforSSM permission.

import json
import boto3
import time
import os

ssm = boto3.client('ssm')
ec2 = boto3.client('ec2')
autoscalingg = boto3.client('autoscaling')

def lambda_handler(event, context):
    # TODO implement
    env = event['environment']
    kafka = event['kafka']
    print (env, kafka)
    status = check_autoscaling_group(env)
    if status:
        add_IP(env, kafka)
        return {
            'body': json.dumps('Function executed, please see the logs!')
        }
    else:
        return {
            'body': json.dumps('The function was not executed, please see the logs!')
        }

def check_autoscaling_group(env):
    response = autoscalingg.describe_auto_scaling_groups(
        AutoScalingGroupNames=[
            'elasticsearch-master-'+str(env)
        ]
    )
    stable = True
    for instance in response['AutoScalingGroups'][0]['Instances']:
        if instance['LifecycleState'] != "InService":
            stable = False
            message = "At least 1 master instance is not in Running status, please wait until all the masters nodes are stable."
            print (message)
            break
    return stable

def add_IP(env, kafka):
    response = ec2.describe_instances(
        Filters=[
            {
                'Name': 'tag:Name',
                'Values': [
                    'elasticsearch-masters-server-'+str(env)
                ]
            },
            {
                'Name': 'instance-state-name',
                'Values': [
                    'running'
                ]
            },
            {
                'Name': 'tag:role',
                'Values': [
                    'master'
                ]
            }
        ]
    )

    for instances in response['Reservations']:
        id = instances['Instances'][0]['InstanceId']
        ip = instances['Instances'][0]['PrivateIpAddress']
        
        #Add ES server IP to Kibana config
        command_to_execute = 'sed -i "s/ELASTICSEARCH_HOSTS: http:\/\/.*/ELASTICSEARCH_HOSTS: http:\/\/'+str(ip)+':9200/g" /home/ubuntu/Kibana/docker-compose.yml'
        execute_in_master(command_to_execute, id, env)
        
        #Add kafka server IP to logstash conf
        kafka = '\\"'+str(kafka)+':9092\\"'
        command_to_execute = 'sed -i "s/bootstrap_servers => .*/bootstrap_servers => ['+str(kafka)+']/g" /home/ubuntu/Logstash/config/logstash/pipeline/my_pipeline.conf'
        execute_in_master(command_to_execute, id, env)
        
        #Add ES server IP to logstash config
        ip = '\\"'+str(ip)+':9200\\"'
        command_to_execute = 'sed -i "s/hosts => .*/hosts => '+str(ip)+'/g" /home/ubuntu/Logstash/config/logstash/pipeline/my_pipeline.conf'
        execute_in_master(command_to_execute, id, env)
        
        #Restart services
        command_to_execute = 'cd /home/ubuntu/Kibana/ && docker-compose up -d'
        execute_in_master(command_to_execute, id, env)
        
        command_to_execute = 'cd /home/ubuntu/Logstash/ && docker-compose up -d'
        execute_in_master(command_to_execute, id, env)
        
        #Just select one master instance
        break
        
def execute_in_master(command_to_execute, id, env):
    print (command_to_execute)
    response = ec2.describe_instances(
        Filters=[
            {
                'Name': 'tag:Name',
                'Values': [
                    'elasticsearch-masters-server-'+str(env)
                ]
            },
            {
                'Name': 'instance-state-name',
                'Values': [
                    'running'
                ]
            },
            {
                'Name': 'tag:role',
                'Values': [
                    'master'
                ]
            }
        ]
    )
    
    for instances in response['Reservations']:
        instance_id = instances['Instances'][0]['InstanceId']
        if instance_id == id:
            response = ssm.send_command(
                InstanceIds=[instance_id],
                DocumentName="AWS-RunShellScript",
                Parameters={'commands': [command_to_execute]}
            )
            time.sleep(1)   

To call a macro inside the cloudformation template you need to add this:

  ModifyInstances:
    Fn::Transform:
        Name: MacroSetUpCluster
        Parameters:
          env: !Ref MyEnv
          kafka: !Ref MyKafkaIP

The stack MacroSetUpCluster needs to have been previously deployed with the lambda function.

Related