Add more_set_headers to Docker image

I am trying to add a module to the base nginx container so I can set some headers. I am using the alpine image so i don't believe i can use apt-get.

This is what I have tried:

FROM nginx:stable-alpine

RUN apk update && \
apk upgrade && \
apk add nginx-mod-http-headers-more

ADD ./nginx/default.conf /etc/nginx/conf.d/default.conf
ADD ./nginx/certs /etc/nginx/certs/self-signed

However if I add the more_set_headers key in the nginx configuration file, this nginx docker image won't start.

and in my docker-compose.yml i have the following:

version: "3.9"

services:
  nginx:
    build:
      context: .
      dockerfile: nginx.dockerfile
    depends_on:
      - php
      - mysql
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./wordpress:/var/www/html:delegated
  mysql:
    image: mysql:latest
    environment:
      MYSQL_DATABASE: wp
      MYSQL_USER: wp
      MYSQL_PASSWORD: secret
      MYSQL_ROOT_PASSWORD: secret
    volumes:
      - ./database/data:/var/lib/mysql
      - ./database/initdb.d:/docker-entrypoint-initdb.d
    ports:
      - "3306:3306"

...