AWS ElasticSearch Logstash 403 Forbidden Access

MySQL and elasticsearch are hosted on aws. And the logstash is running on an ec2. I am not using a VPC.I can connect to MySQL locally or on my ec2.

Modifying question a bit. Here is the new logstash file in my ec2. My es2 instance is not SSL certified, is that a problem?

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://aws.xxxxx.us-east-1.rds.amazonaws.com:3306/stuffed?user=admin&password=pword"
        jdbc_user => "admin"
        jdbc_password => "pword"
        schedule => "* * * * *"
        jdbc_validate_connection => true
        jdbc_driver_library => "mysql-connector-java-8.0.19.jar"
        jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
        statement => "SELECT * from foo"
        type => "foo"
        tags => ["foo"]
    }

    jdbc {
        jdbc_connection_string => "jdbc:mysql://aws.xxxxx.us-east-1.rds.amazonaws.com:3306/stuffed?user=admin&password=pword"
        jdbc_user => "admin"
        jdbc_password => "pword"
        schedule => "* * * * *"
        jdbc_validate_connection => true
        jdbc_driver_library => "mysql-connector-java-8.0.19.jar"
        jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
        statement => "SELECT * from cat"
        type => "cat"
        tags => ["cat"]
    }

    jdbc {
        jdbc_connection_string => "jdbc:mysql://aws.xxxxx.us-east-1.rds.amazonaws.com:3306/stuffed?user=admin&password=pword"
        jdbc_user => "admin"
        jdbc_password => "pword"
        schedule => "* * * * *"
        jdbc_validate_connection => true
        jdbc_driver_library => "mysql-connector-java-8.0.19.jar"
        jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
        statement => "SELECT * from rest"
        type => "rest"
        tags => ["rest"]
    }
}
output {
    stdout { codec => json_lines }

    if "foo" in [tags] {
        amazon_es {
            hosts => ["https://es1.stuffed-es-mysql.us-east-1.es.amazonaws.com"]
            index => "foo"
            region => "us-east-1"
        aws_access_key_id => "id"
        aws_secret_access_key => "key"
            document_type => "foo-%{+YYYY.MM.dd}"
        }
    }

    if "cat" in [tags] {
        amazon_es {
            hosts => ["https://es1.stuffed-es-mysql.us-east-1.es.amazonaws.com"]
            index => "cat"
            region => "us-east-1"
        aws_access_key_id => "id"
        aws_secret_access_key => "key"
            document_type => "cat-%{+YYYY.MM.dd}"
        }
    }

    if "rest" in [tags] {
        amazon_es {
            hosts => ["https://es1.stuffed-es-mysql.us-east-1.es.amazonaws.com"]
            index => "rest"
            region => "us-east-1"
        aws_access_key_id => "id"
        aws_secret_access_key => "key"            
        document_type => "rest-%{+YYYY.MM.dd}"
        }
    }
}

Now the issue I'm getting is a 403 forbidden error.

I did create a user in AWS with AmazonESFullAccess (AWS managed policy) permission. I'm not sure what else to do anymore. I am not using a VPC, I was trying to avoid that. So I want to stick with public access.

I tried creating new ElasticSearch Service based off this guide: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html

but I get an error error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://user:xxxxxx@mysql-abcdefghijkl.us-east-1.es.amazonaws.com:9200/]

and the logstash ouput for this instance is:

elasticsearch {
        hosts => ["https://es-mysql.us-east-1.es.amazonaws.com/"]
        index => "category"
        user => "user"
        password => "password"
        document_type => "cat-%{+YYYY.MM.dd}"
    }

Obviously this isn't the preferred method, but I'm really just trying to setup a dev/personal environment.

Also, I am able to login into Kibana with this instance.

ACCESS POLICY

For the first elastic search service:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111:user/root"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:11111111:domain/stuffed-es-mysql/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:166216189490:domain/stuffed-es-mysql/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "11.11.11.111",
            "111.111.1.111",
            "111.111.1.1",
            "1.11.111.111",
          ]
        }
      }
    }
  ]
}

For the second ElasticSearch Service I created:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:xxxxxxxxx:domain/es-mysql/*"
    }
  ]
}