Docker build is not using cache all the time for react/node project

I have a docker file for deploying my react project. Here is the dockerfile:

FROM node:14.2.0

COPY package.json /tmp/package.json
RUN cd /tmp && npm install --silent
RUN mkdir -p /home/node/app/ && cp -a /tmp/node_modules /home/node/app/

WORKDIR /home/node/app/
USER root
COPY . ./

I believe if there is no change in package.json it will use cache for npm install(2nd step) and copying node_modules (3rd step).

But even if there is no change in package.json it does not. How do I cache the steps then ?