Exclude file in Docker COPY and but use it in another COPY

My folder structure looks like:

src/
public/
.env
package.json
nginx.conf

My Dockerfile looks like this:

FROM node:14 as builder                   # line-1
WORKDIR /app                              # line-2
COPY . /app                               # line-3 nginx.conf is copied but not needed
RUN yarn && yarn build                    # line-4

FROM nginx:1.21.1                         # line-5
WORKDIR /app                              # line-6
COPY --from=builder /app/build /app/build # line-7
COPY nginx.conf /app                      # line-8 nginx.conf is copied

CMD nginx -c /app/nginx.conf              # line-9

When I change nginx.conf docker layer caching becomes invalidated because of the 3rd line.

I cannot use .dockerignore because I need this file.

Obvious solution is to copy all the needed folders one by one, but that seems to be unreliable because project structure can change from time to time and I'd like to avoid extra effort of Dockerfile maintenance & updating.

Is there shorter way to overcome this apart from that is mentioned here?