I'm trying to create a stack on AWS CloudFormation, with an EC2 instance and 2 S3 buckets. My script is attempting to assign a Policy to the EC2 instance that allows access to the Storage bucket, but no matter what I do the rights are not assigned. Additionally, the userdata is not executed at all.
I tried testing thoroughly if the EC2 really does not have the rights: the CLI confirmed that it does not. I replaced the userdata with a simple script making a textfile, it really is not created. AWS Designer gives no complaints and shows the correct template structure. The stack description runs and executes with no errors, except the S3 storage bucket access and the user data don't work (no warnings).
After a LOT of manual editing and checking very carefully with the documentation, I realised I should have done this in a higher level language. Therefore I tried to import the script in a simple python Troposphere script using the templateGenerator. This leads to the following error (no other errors are created anywhere so far, everything just silently goes wrong, JSON syntax validators also have no complaints):
TypeError: <class 'troposphere.iam.PolicyType'>: MickStorageS3BucketsPolicy.PolicyDocument is <class 'list'>, expected (<class 'dict'>,)
However, clearly my PolicyDocument is of type dictionary, and I don't understand how it can be interpreted as a list. I have stared at this for many hours now, I may have become blind to the problem but I would Really really appreciate any help at this point!!!!
The security group and inbound traffic settings do work properly, my dockerized flask app runs fine (on the EC2) but just can't access the bucket (though I have to start it manually through SSH because userdata won't execute, I also tried doing that using the CFN-init segment in the ec2 metadata (under commands) but nothing executes, even if I try to run CFNinit manually after connecting by SSH).
This is the cloudformation template I wrote:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Attach IAM Role to an EC2",
"Parameters" : {
"KeyName" : {
"Description" : "EC2 Instance SSH Key",
"Type" : "AWS::EC2::KeyPair::KeyName",
"Default" : "MickFirstSSHKeyPair"
},
"InstanceType" : {
"Description" : "EC2 instance specs configuration",
"Type" : "String",
"Default" : "t2.micro",
"AllowedValues" : ["t2.micro", "t2.small", "t2.medium"]
}
},
"Mappings" : {
"AMIs" : {
"us-east-1" : {
"Name" : "ami-8c1be5f6"
},
"us-east-2" : {
"Name" : "ami-c5062ba0"
},
"eu-west-1" : {
"Name" : "ami-acd005d5"
},
"eu-west-3" : {
"Name" : "ami-05b93cd5a1b552734"
},
"us-west-2" : {
"Name" : "ami-0f2176987ee50226e"
},
"ap-southeast-2" : {
"Name" : "ami-8536d6e7"
}
}
},
"Resources" : {
"mickmys3storageinstance" : {
"Type" : "AWS::S3::Bucket",
"Properties" : {
}
},
"mickmys3processedinstance" : {
"Type" : "AWS::S3::Bucket",
"Properties" : {
}
},
"MickMainEC2" : {
"Type" : "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
},
"commands" : {
}
}
}
},
"Properties" : {
"UserData": {
"Fn::Base64" : "echo 'Heelo ww' > ~/hello.txt"
},
"InstanceType" : {
"Ref" : "InstanceType"
},
"ImageId" : {
"Fn::FindInMap" : [
"AMIs",
{
"Ref" : "AWS::Region"
},
"Name"
]
},
"KeyName" : {
"Ref" : "KeyName"
},
"IamInstanceProfile" : {
"Ref" : "ListS3BucketsInstanceProfile"
},
"SecurityGroupIds" : [
{
"Ref" : "SSHAccessSG"
},
{
"Ref" : "PublicAccessSG"
}
],
"Tags" : [
{
"Key" : "Name",
"Value" : "MickMainEC2"
}
]
}
},
"SSHAccessSG" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Allow SSH access from anywhere",
"SecurityGroupIngress" : [
{
"FromPort" : "22",
"ToPort" : "22",
"IpProtocol" : "tcp",
"CidrIp" : "0.0.0.0/0"
}
],
"Tags" : [
{
"Key" : "Name",
"Value" : "SSHAccessSG"
}
]
}
},
"PublicAccessSG" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Allow HTML requests from anywhere",
"SecurityGroupIngress" : [
{
"FromPort" : "80",
"ToPort" : "80",
"IpProtocol" : "tcp",
"CidrIp" : "0.0.0.0/0"
}
],
"Tags" : [
{
"Key" : "Name",
"Value" : "PublicAccessSG"
}
]
}
},
"ListS3BucketsInstanceProfile" : {
"Type" : "AWS::IAM::InstanceProfile",
"Properties" : {
"Path" : "/",
"Roles" : [
{
"Ref" : "MickListS3BucketsRole"
}
]
}
},
"MickStorageS3BucketsPolicy" : {
"Type" : "AWS::IAM::Policy",
"Properties" : {
"PolicyName" : "MickStorageS3BucketsPolicy",
"PolicyDocument" : {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::mickmys3storageinstance", "arn:aws:s3:::mickmys3storageinstance/*"
]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": ["s3:*Object"],
"Resource": [
"arn:aws:s3:::mickmys3storageinstance", "arn:aws:s3:::mickmys3storageinstance/*"
]
}
]
},
"Roles" : [
{
"Ref" : "MickListS3BucketsRole"
}
]
}
},
"MickListS3BucketsRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Principal" : {
"Service" : ["ec2.amazonaws.com"]
},
"Action" : [
"sts:AssumeRole"
]
}
]
},
"Path" : "/"
}
}
},
"Outputs" : {
"EC2" : {
"Description" : "EC2 IP address",
"Value" : {
"Fn::Join" : [
"",
[
"ssh ec2-user@",
{
"Fn::GetAtt" : [
"MickMainEC2",
"PublicIp"
]
},
" -i ",
{
"Ref" : "KeyName"
},
".pem"
]
]
}
}
}
}
Here is my troposphere script generating the error on importing the above:
from troposphere import Ref, Template
import troposphere.ec2 as ec2
from troposphere.template_generator import TemplateGenerator
import json
with open("myStackFile.JSON") as f:
json_template = json.load(f)
template = TemplateGenerator(json_template)
template.to_json()
print(template.to_yaml())
I expected the roles to be assigned correctly, as well as the userdata to be executed. I expected troposphere to import the JSON, as it has the correct syntax and also the correct class typing as according to the documentation as far as I can see. I have doublechecked everything by hand for many many hours, I am not sure how to proceed finding the issue with this CloudFormation script. In the future (and I would advise anyone to do the same) I will not edit JSON (or worse, YAML) files by hand any more, and use higher level tools exclusively.
Thank you for ANY help/pointers!
Kind regards