This answer is possibly still incomplete, but fascinated by the problem, I have taken some time to debug it using the following Dockerfile:
FROM debian:10
SHELL ["/bin/sh", "-ec"]
RUN apt-get update > /dev/null; apt-get -y install psmisc > /dev/null
# First layer is good: expected 1, got 1
RUN /bin/bash -exc "pstree; echo SHLVL=\$SHLVL"
# The surprising example (getting the escaping right is already tricky)
RUN /bin/bash -exc "/bin/bash -exc \"pstree; echo SHLVL=\\\$SHLVL\"" # expected 2, got 1
RUN /bin/bash -exc "/bin/bash -exc \"/bin/bash -exc \\\"pstree; echo SHLVL=\\\\\\\$SHLVL\\\"\"" # expected 3, got 1
# Now what happens if two commands run in the inner bash
RUN /bin/bash -exc ":; /bin/bash -exc \":; pstree; echo SHLVL=\\\$SHLVL\"" # expected 2, got 1
RUN /bin/bash -exc ":; /bin/bash -exc \":; /bin/bash -exc \\\":; pstree; echo SHLVL=\\\\\\\$SHLVL\\\"\"" # expected 3, got 1
The interesting thing seems to be: In case a bash invocation is followed directly by another one, it will be "optimized" (?) away. As far as I can tell, this is not a Docker-specific thing, because it can be reproduced interactively (on my Debian 10 system, the interactive command sequence from the question produces 1, 2, 2 and not 1, 2, 3 for the nesting!).
In any case, the output from building the Dockerfile is as follows:
Sending build context to Docker daemon 30.21kB
Step 1/8 : FROM debian:10
---> 8e9f8546050d
Step 2/8 : SHELL ["/bin/sh", "-ec"]
---> Running in 3509ef249c45
Removing intermediate container 3509ef249c45
---> 8956c1fddb7c
Step 3/8 : RUN apt-get update > /dev/null; apt-get -y install psmisc > /dev/null
---> Running in 5cabec19144a
debconf: delaying package configuration, since apt-utils is not installed
Removing intermediate container 5cabec19144a
---> 64cad97f7793
Step 4/8 : RUN /bin/bash -exc "pstree; echo SHLVL=\$SHLVL"
---> Running in 22a0aa663163
+ pstree
sh---bash---pstree
+ echo SHLVL=1
SHLVL=1
Removing intermediate container 22a0aa663163
---> 4caa146b24f6
Step 5/8 : RUN /bin/bash -exc "/bin/bash -exc \"pstree; echo SHLVL=\\\$SHLVL\"" # expected 2, got 1
---> Running in 538ff45db230
+ /bin/bash -exc 'pstree; echo SHLVL=$SHLVL'
+ pstree
sh---bash---pstree
SHLVL=1
+ echo SHLVL=1
Removing intermediate container 538ff45db230
---> 1d4c9c2638fa
Step 6/8 : RUN /bin/bash -exc "/bin/bash -exc \"/bin/bash -exc \\\"pstree; echo SHLVL=\\\\\\\$SHLVL\\\"\"" # expected 3, got 1
---> Running in 3f0650d4d21b
+ /bin/bash -exc '/bin/bash -exc "pstree; echo SHLVL=\$SHLVL"'
+ /bin/bash -exc 'pstree; echo SHLVL=$SHLVL'
+ pstree
sh---bash---pstree
SHLVL=1
+ echo SHLVL=1
Removing intermediate container 3f0650d4d21b
---> 2d977033884d
Step 7/8 : RUN /bin/bash -exc ":; /bin/bash -exc \":; pstree; echo SHLVL=\\\$SHLVL\"" # expected 2, got 1
---> Running in 39b79af0f558
+ :
+ /bin/bash -exc ':; pstree; echo SHLVL=$SHLVL'
+ :
+ pstree
sh---bash---bash---pstree
+ echo SHLVL=2
SHLVL=2
Removing intermediate container 39b79af0f558
---> 48170e9bcb01
Step 8/8 : RUN /bin/bash -exc ":; /bin/bash -exc \":; /bin/bash -exc \\\":; pstree; echo SHLVL=\\\\\\\$SHLVL\\\"\"" # expected 3, got 1
---> Running in 456e6ec421ca
+ :
+ /bin/bash -exc ':; /bin/bash -exc ":; pstree; echo SHLVL=\$SHLVL"'
+ :
+ /bin/bash -exc ':; pstree; echo SHLVL=$SHLVL'
+ :
+ pstree
sh---bash---bash---bash---pstree
+ echo SHLVL=3
SHLVL=3
Removing intermediate container 456e6ec421ca
---> 30a07d3bdc95
Successfully built 30a07d3bdc95
Successfully tagged test:latest
Finally, the pstree output is interesting because it shows how there is actually no other bash process running at the respective points (i.e. the variable tracks the actual shell nesting correctly in all of the cases, it is just that sometimes there are less shells running than expected).
