AWS CodeDeploy with Lambda Layer

I have a template.yaml function where I am defining a simple lambda function and a simple lambda layer.

I am able to deploy it properly using sam build, sam package and sam deploy and it works just fine. But I am trying to deploy using CodePipeline (Source -> Build -> Prod). The prod stage works when I don't add the lambda layer definition. It does not work when I add it.

The issue is the lambda layer references .aws-sam directory which is only created after I would do a sam build. I am unsure how to reference the function to create a lambda layer. I saw that CodeBuild outputs a packaged.yaml file in s3 but this is the code URIs:

Func1: CodeUri: s3://{}/{hash}

but the layer uri is not changing to point to the correct uri: ContentUri: ./.aws-sam/build/ImageProcessingLib

Here is the template.yaml definition that works without CodePipeline.

  ImageProcessingLib:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./src/layers/imageProcessingLib/
      Handler: image_processing_lib.lambda_handler
      Runtime: python3.7


  ImageProcessingLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      Description: Lambda layer for writing additional functions for indexing and searching images
      LayerName: ImageProcessingLib
      ContentUri: ./.aws-sam/build/ImageProcessingLib
      CompatibleRuntimes: 
        - python3.7
      RetentionPolicy: Delete
    DependsOn: ImageProcessingLib

Is the fix as simple as changing the CodeUri of ImageProcessingLayer?