Referencing bucket name as variable to a resource in s3 event in serverless.yml

Question

I have the following configuration in serverless.yml functioning well:

service: fx-crawler-av
frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
  region: eu-west-1

  environment:
    S3BucketName: !Ref 'S3BucketFXDataStorage'
    AlphaVantageAPIKey: ${ssm:AlphaVantageAPIKey}

package:
  exclude:
    - requirements.txt
    - package.json
    - package-lock.json
    - node_modules/**
    - .serverless/**
    - .env/**

functions:
  fetch_api:
    handler: handler.fetch_api
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - "s3:PutObject"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"
              - "/*"
  set_ccy_pair_scope:
    handler: handler.set_ccy_pair_scope
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - "s3:GetObject"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"
              - "/*"
      - Effect: "Allow"        
        Action:
          - "s3:ListBucket"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"
  get_ccy_list:
    handler: handler.get_ccy_list
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - "s3:GetObject"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"
              - "/*"
  append_parquet_file:
    handler: handler.append_parquet_file
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - "s3:GetObject"
          - "s3:PutObject"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"
              - "/*"
      - Effect: "Allow"        
        Action:
          - "s3:ListBucket"
        Resource:
          Fn::Join:
            - ""
            - - "arn:aws:s3:::"
              - "Ref": "S3BucketFXDataStorage"

resources:
  Resources:
    S3BucketFXDataStorage:
      Type: 'AWS::S3::Bucket'
      Properties:
        BucketName: !Join
          - ''
          - - !Ref 'AWS::AccountId'
            - '-fx-data-storage'

stepFunctions:
  stateMachines:
    FXCrawler:
      events:
        - schedule: cron(35 21 * * ? *)
      definition:
        Comment: "State machine for crawling AlphaVantage API for FX rates"
        StartAt: GetCurrencyList
        States:
          GetCurrencyList:
            Type: Task
            Resource: 
              Fn::GetAtt: [get_ccy_list, Arn]
            Next: SetCurrencyPairScope
          SetCurrencyPairScope:
            Type: Task
            Resource:
              Fn::GetAtt: [set_ccy_pair_scope, Arn]
            Next: mapped_task
          mapped_task:
            Type: Map
            ItemsPath: '$.ccy_pair_scope'
            MaxConcurrency: 1
            Iterator:
              StartAt: FetchAPI
              States:
                FetchAPI:
                  Type: Task
                  Resource:
                    Fn::GetAtt: [fetch_api, Arn]
                  Next: WaitForAPI
                WaitForAPI:
                  Type: Wait
                  Seconds: 60
                  End: true
            End: true

plugins:
  - serverless-python-requirements
  - serverless-iam-roles-per-function
  - serverless-step-functions

custom:
  pythonRequirements:
    slim: true
    dockerizePip: true
    useDownloadCache: true

Now I would like to add an S3 event trigger to my append_parquet_file function, e.g.:

  append_parquet_file:
    handler: handler.append_parquet_file
    events:
      - s3:
          bucket: !Ref 'S3BucketFXDataStorage'
          event: s3:ObjectCreated:*
          rules:
            - suffix: .json

But unfortunately this does not seem to work..

I get a warning during deployment Serverless: Configuration warning at 'functions.append_parquet_file.events[0].s3.bucket': should be string
The deployment fails:

Serverless Error ---------------------------------------
 
  [object Object] - Bucket name must conform to pattern (?!^(\d{1,3}\.){3}\d{1,3}$)(^(([a-z0-9]|[a-z0-9][a-z0-9-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9-]*[a-z0-9])$). Please check provider.s3.[object Object] and/or s3 events of function "append_parquet_file".

I tried multiple ways to create and inject the bucket name as variable but did not succeed. Any idea what can go wrong?

user_8352921 · Answer

You don't need to create bucket resouces, according to the docs, you can specify the bucket and add properties you need and bucket will create and configure automatically to setup the trigger

functions:
  resize:
    handler: resize.handler
    events:
      - s3: bucketOne
 
provider:
  s3:
    bucketOne:
      name: my-custom-bucket-name
      # Eventual additional properties in camel case

https://www.serverless.com/framework/docs/providers/aws/events/s3#custom-bucket-configuration

user_248823 · Answer

You should reference your bucket as follows:

bucket: ${self:resources.Resources.S3BucketFXDataStorage.Properties.BucketName}

user_4232386 · Answer

All you need is this: (remove the double quotes)

    Resource: !Ref S3BucketFXDataStorage
    event: s3:ObjectCreated:*
    existing: true

and you have another problem in the policies, try this instead to reference you bucket in the policies:

Resource: !Join ["", [!GetAtt S3BucketFXDataStorage.Arn, "/*" ]]

Referencing bucket name as variable to a resource in s3 event in serverless.yml

3 Answers