Reuse steps in docker build image. Cache?

I am trying to increase the speed of my CI/CD. One of the steps is to build my docker image. After the image is built I push it to my registry in order to make pull later in the next CI/CD iteration||loop||run before starting build process in order to reuse docker build image layer cache. It is expected this step to take some time, however, it can be reduced if previous images have been built before. The docker should not repeat (or at least this is what I expect) the same layer (Dockerfile line) if the lines before do not change.

Following real example:

I have this directory

ls  
Dockerfile  
somefile.txt

Dockerfile content:

cat Dockerfile
FROM ruby:2.7.4-slim

ENTRYPOINT irb

When I build first time:

docker build -t my-registry/my-docker-cache .
Sending build context to Docker daemon   2.56kB
Step 1/2 : FROM ruby:2.7.4-slim
 ---> db4073acbaac
Step 2/2 : ENTRYPOINT irb
 ---> Running in 14055ed3e5d1
Removing intermediate container 14055ed3e5d1
 ---> f4f317dde34d
Successfully built f4f317dde34d
Successfully tagged my-registry/my-docker-cache:latest

Everything is like expected, no cache, every step is executed. If I repeat the command:

docker build -t my-registry/my-docker-cache .
Sending build context to Docker daemon   2.56kB
Step 1/2 : FROM ruby:2.7.4-slim
 ---> db4073acbaac
Step 2/2 : ENTRYPOINT irb
 ---> Using cache
 ---> f4f317dde34d
Successfully built f4f317dde34d
Successfully tagged my-registry/my-docker-cache:latest

Everything was fine and as it is expected Step 2 says Using cache and it is not executed.

I will push my image to the registry:

docker push my-registry/my-docker-cache
aaf6670012a0: Mounted from previous-image-in-other-experiments 
e21a639a3286: Mounted from previous-image-in-other-experiments
f01aae87d116: Mounted from previous-image-in-other-experiments
04f100f96a69: Mounted from previous-image-in-other-experiments
e1bbcf243d0e: Mounted from previous-image-in-other-experiments
latest: digest: sha256:317cbcbb751fc90e3656818a7942ddd67fb8cb9c24841f57cce272bc47c02ce5 size: 1366

Now I will delete the image from my local docker:

docker rmi my-registry/my-docker-cache
Untagged: my-registry/my-docker-cache:latest
Untagged: my-registry/my-docker-cache@sha256:317cbcbb751fc90e3656818a7942ddd67fb8cb9c24841f57cce272bc47c02ce5
Deleted: sha256:f4f317dde34ddba589867c3d81af9cd6fe30732afc0173a53740c271d2b70ed3

Now I will pull from registry:

docker pull my-registry/my-docker-cache
Using default tag: latest
latest: Pulling from my-docker-cache
eff15d958d66: Already exists 
923e91ae3a1b: Already exists 
2aa5d3a4a151: Already exists 
bc64adf2d0b2: Already exists 
bfc5cca7d80e: Already exists 
Digest: sha256:317cbcbb751fc90e3656818a7942ddd67fb8cb9c24841f57cce272bc47c02ce5
Status: Downloaded newer image for my-registry/my-docker-cache:latest
my-registry/my-docker-cache:latest

Now I will build the image again even with the same name:

docker build -t my-registry/my-docker-cache .
Sending build context to Docker daemon   2.56kB
Step 1/2 : FROM ruby:2.7.4-slim
 ---> db4073acbaac
Step 2/2 : ENTRYPOINT irb
 ---> Running in b779e2e702bf
Removing intermediate container b779e2e702bf
 ---> 73464e25f559
Successfully built 73464e25f559
Successfully tagged my-registry/my-docker-cache:latest

I was expecting Step 2 say Using cache and do not execute. In this case ENTRYPOINT step run fast but in case of RUN bundle install or RUN apt update; apt install bla bla bla is not the same.

What I am doing wrong, or what can I do for reaching my goal?

docker version
Client: Docker Engine - Community
 Cloud integration: v1.0.28
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:57 2022
 OS/Arch:           linux/amd64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Desktop 4.11.0 (83626)
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0